system

Abstract

A system is provided.SOLUTION: A system including means for collecting a user's speech with a voice input device, means for converting the collected voice data into character data, means for analyzing the character data using natural language processing (NLP) to determine the user's needs, means for reinforcing the analysis result in cooperation with other sensor information, means for identifying a required article and generating and transmitting a suggestion message to the user, and means for executing an order to an EC site or an online supermarket based on the user's approval.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It is important to reduce the effort required for users to frequently order items necessary for daily life. However, current systems require users to manually check the items and place the order, which is time-consuming and labor-intensive. Furthermore, there is a lack of a mechanism for quickly and accurately responding to items that users tend to forget or need urgently. To solve these problems, there is a need for a system that automatically orders items by utilizing users' daily conversations and tweets. [Means for solving the problem]

[0005] The system of the present invention solves the above problems by comprising the following means: a means for collecting user conversations using a voice input device, a means for converting the collected voice data into text data, and a means for analyzing the text data using natural language processing (NLP) to determine the user's needs. It also includes a means for augmenting the analysis results in conjunction with other sensor information, and a means for identifying necessary items and generating and sending a suggested message to the user. If the suggested message is approved by the user, a means for placing an order on an e-commerce site or online supermarket is provided. Furthermore, a means for proposing fees based on dynamic pricing makes it possible to analyze the user's emotions and urgency from their tweets and conversations. In this way, a system is realized that quickly and automatically secures necessary items based on the user's daily tweets.

[0006] "User" refers to any individual or entity that uses the System.

[0007] "Conversation" refers to the spoken language that a user utters on a daily basis.

[0008] "Audio input device" refers to a device for collecting a user's voice, including a microphone.

[0009] "Voice Data" means a digital representation of a user's voice collected by an audio input device.

[0010] "Text data" refers to data that has been converted from voice data into text format using voice recognition technology.

[0011] "Natural Language Processing (NLP)" refers to the techniques and processes that enable computers to understand, analyze, and generate human language.

[0012] "Sensor Information" refers to data obtained from various sensors that provide additional information such as temperature, humidity, and inventory status.

[0013] A "suggestion message" is a notification or message generated to inform a user of a suggested item they need.

[0014] "EC site" refers to a website for selling and purchasing products over the Internet.

[0015] "Online supermarket" refers to a website or application that provides services that allow users to purchase groceries and daily necessities online.

[0016] "Ordering" refers to the process of purchasing an item that the user has approved based on the suggested message.

[0017] "Dynamic pricing" is a pricing method that changes prices based on supply, demand, and other conditions.

[0018] "Fees" refers to the fees collected from users for providing services.

[0019] A "tweet" refers to a short spoken expression casually uttered by a user. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system that utilizes user conversations and tweets to automatically order necessary items. This system collects and analyzes the user's voice, identifies the necessary items, makes ordering suggestions to the user, and, if the user approves, automatically places the order with an online supermarket or e-commerce site.

[0042] System configuration

[0043] 1. Voice Input Device

[0044] The terminal is equipped with a built-in voice input device equipped with a microphone, which constantly monitors the user's everyday conversations and tweets.

[0045] 2. Voice Recognition Technology

[0046] The server receives the voice data sent from the voice input device and converts the voice data into text data using voice recognition technology.

[0047] 3. Natural Language Processing (NLP)

[0048] The server analyzes the text data using natural language processing (NLP) to understand and identify the user's needs and requirements.

[0049] 4. Linking with sensor information

[0050] The server will then integrate other sensor information (e.g., inventory sensors and temperature and humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs.

[0051] 5. Creating and sending a proposal message

[0052] The server identifies the items the user needs based on the analysis results, generates a suggestion message for the user, and sends it to the user's device. The suggestion message is then delivered to the user via a smartphone notification or voice command.

[0053] 6. User Authorization and Automated Orders

[0054] The terminal receives the user's approval and sends that approval information to the server, which then uses the API of the online supermarket or e-commerce site to automatically place the order based on that information.

[0055] Specific operation example

[0056] For example, consider the case where a user tweets one day, "I'm running out of milk..."

[0057] 1. Audio collection

[0058] A user says, "I'm running out of milk..."

[0059] The device collects this audio and sends it to the server.

[0060] 2. Voice Recognition

[0061] The server converts the audio data into text data.

[0062] The text data obtained is "We're running out of milk..."

[0063] 3. Natural Language Processing

[0064] The server uses natural language processing to analyze the text and extract the need, "I need milk."

[0065] 4. Linking with sensor information

[0066] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[0067] 5. Creating and sending a proposal message

[0068] The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and sends it to the user's smartphone.

[0069] 6. User Authorization and Order Execution

[0070] The user checks the notification on their smartphone and responds verbally, "Yes, please place the order."

[0071] The device sends this approval to the server.

[0072] The server automatically executes the milk order using the online supermarket's API.

[0073] In this way, by using the system of the present invention, users can automatically order the items they need based on casual daily tweets, significantly reducing the effort required for shopping.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] The voice input device constantly monitors the user's daily conversations and tweets. When voice data is input, the device collects the voice data and starts recording.

[0077] Step 2:

[0078] The device sends the collected voice data to the server in real time, and the voice data is securely transmitted over the network.

[0079] Step 3:

[0080] The server converts the received voice data into text data using a voice recognition engine, which uses a voice recognition algorithm.

[0081] Step 4:

[0082] The server passes the text data to a natural language processing (NLP) framework for analysis, which involves tokenizing the text into words, extracting nouns and verbs, and performing contextual analysis.

[0083] Step 5:

[0084] The server determines the user's intentions and needs from the analysis results. For example, if someone tweets "I'm running out of milk...", it will recognize that "I need milk."

[0085] Step 6:

[0086] The server checks the user's smart device and IoT sensor information, for example, checking the refrigerator's internal camera and inventory sensor to see if there is a low level of milk.

[0087] Step 7:

[0088] The server augments the analysis with sensor information and ultimately identifies the items needed. This specific step verifies that the sensor readings match expectations.

[0089] Step 8:

[0090] The server identifies the items needed and generates a suggestion message based on that, for example, "You're running low on milk. Would you like to order some?"

[0091] Step 9:

[0092] The server sends the generated proposal message to the user's terminal via a transmission protocol. The terminal notifies the user of the proposal message and waits for a response from the user.

[0093] Step 10:

[0094] The user reviews the suggestion message and takes an action to accept or reject it, for example, by saying "Yes, please place the order."

[0095] Step 11:

[0096] The device sends the user's authorization information to the server, using secure communication methods.

[0097] Step 12:

[0098] The server receives the authorization information and automatically orders the identified items using the API of the e-commerce site or online supermarket. Specifically, it generates an order request and sends it to the API endpoint.

[0099] Step 13:

[0100] The server retrieves the order confirmation and notifies the user that the order has been completed, generating detailed notifications along the way, including success or failure status.

[0101] This series of steps realizes a process for quickly and efficiently automatically ordering the necessary items from the user's spontaneous tweets.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] Conventional e-commerce systems require users to manually order the items they need each time, which is time-consuming and often leads to users forgetting to order. It is also difficult to grasp the user's situation and needs in real time and automate appropriate suggestions and orders based on that information. There is a need for a system that can solve these problems and enable users to purchase goods more conveniently and efficiently.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, and means for analyzing the text data using natural language processing (NLP) to determine the user's needs. This enables the server to automatically determine the items needed from the user's tweets and conversations, generate and send suggestion messages, and enable efficient purchasing.

[0107] "User" refers to an individual or business entity that uses the system.

[0108] "Audio input device" refers to a hardware device with a built-in microphone that collects a user's voice.

[0109] "Voice Data" refers to digitally recorded data of a user's voice collected by a voice input device.

[0110] "Character data" refers to text-format data converted from voice data using voice recognition technology.

[0111] "Natural Language Processing (NLP)" refers to the technology of analyzing text data to understand and identify user needs and requirements.

[0112] "Sensor information" refers to data obtained from IoT devices such as inventory sensors and temperature / humidity sensors inside the refrigerator.

[0113] A "suggestion message" refers to a message that is generated based on the analysis results and that suggests to the user to purchase an item.

[0114] "E-commerce system" refers to an online platform that enables the purchase of goods and services via the Internet.

[0115] "Dynamic pricing" refers to a pricing method in which prices fluctuate according to supply and demand.

[0116] "Approval information" refers to data provided by a user to indicate approval of a proposed message.

[0117] This invention is a system that automatically orders necessary items by utilizing user conversations and tweets. This system consists of a voice input device, i.e., a terminal equipped with a microphone for collecting the user's voice, and a server for processing this voice.

[0118] The device is equipped with a voice input device such as a smart speaker, which constantly collects the user's everyday conversations and tweets. For example, if a user tweets, "I'm running out of milk...", the voice data is encrypted and sent to a server in real time. The server then converts the voice data into text data using voice recognition technology such as Google® Speech-to-Text API or Azure® Speech-to-Text.

[0119] The server then analyzes this text data using natural language processing (NLP) techniques such as BERT. This analysis determines the user's specific needs and requests, such as "I need milk." The server then checks the refrigerator's inventory, temperature, and humidity sensors to reinforce the analysis. Based on this information, the server determines whether the item the user needs is actually in short supply.

[0120] Once the server identifies the need, it generates a suggestion message saying, "You're low on milk. Would you like to order some?" This message is sent to the user via their smartphone or smart speaker. When the user responds to this suggestion with "Yes, please order," the device collects the voice and sends it back to the server.

[0121] Once the server confirms the user's approval, it uses the API of the e-commerce system to automatically place the order. For example, it automates the ordering process at online supermarkets and e-commerce sites, and payments are made using PayPal or credit card information. Once the order is complete, a confirmation email is sent to the user.

[0122] Specific examples

[0123] Example 1: When you are running out of milk

[0124] 1. A user tweets, "I'm running out of milk..."

[0125] 2. The device collects the audio and sends it to the server.

[0126] 3. The server uses the Google Speech-to-Text API to convert the audio data into text data.

[0127] 4. The server uses BERT to extract the need "I need milk."

[0128] 5. The server retrieves data from the refrigerator's inventory sensor and checks for low inventory.

[0129] 6. The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and sends it to the smartphone.

[0130] 7. The user responds, "Yes, place the order."

[0131] 8. The device sends the authorization information to the server.

[0132] 9. The server automatically orders milk using the API of the e-commerce system.

[0133] This system allows users to automatically order the items they need based on casual daily tweets, significantly reducing the effort required for shopping.

[0134] Examples of prompt statements

[0135] "Please tell me how to collect tweets about how I'm running out of milk, determine their needs using natural language processing, and send them suggestion messages."

[0136] "Please tell me the specific steps of a system that uses voice recognition technology to identify the items needed from the user's voice and automatically places an order with an online supermarket."

[0137] As described above, the present invention utilizes users' conversations and tweets, and makes use of voice input devices and advanced analysis technology to make users' lives more convenient.

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] Step 1: Collect audio input

[0140] The device uses a built-in microphone to constantly collect the user's everyday conversations and tweets. The input is the user's voice, which is converted into digital voice data. Specifically, when a user tweets, "I'm running out of milk...", the device collects the voice in real time. The output is the collected voice data.

[0141] Step 2: Sending audio data

[0142] The voice data collected by the terminal is encrypted and sent to the server via the Internet. The input is the digital voice data collected in step 1, and the output is the voice data sent to the server. Specifically, the terminal converts the collected voice data into packets and sends them to the server.

[0143] Step 3: Voice Recognition

[0144] The server uses speech recognition technology such as Google Speech-to-Text API or Azure Speech-to-Text to convert the voice data into text data. The input is the voice data sent in step 2, and the output is text data. Specifically, the server analyzes the voice data and generates text data such as "I'm running low on milk..."

[0145] Step 4: Natural Language Processing (NLP)

[0146] The server uses natural language processing (NLP) techniques such as BERT to analyze the text data and identify the user's needs. The input is the text data generated in step 3, and the output is the user's identified need (e.g., "I need milk"). Specifically, the server analyzes the text data and extracts the need "I need milk" from "I'm running low on milk..."

[0147] Step 5: Acquire sensor information

[0148] The server obtains data from the inventory sensor and other sensors in the refrigerator to augment the analysis results. The input is data from the inventory sensor and temperature / humidity sensors in the refrigerator, and the output is the confirmed inventory status (e.g., "Milk is low in stock"). Specifically, the server obtains information that "Milk is low in stock" from the inventory sensor in the refrigerator.

[0149] Step 6: Proposal message generation

[0150] The server generates a suggestion message for the user based on the analysis results and sensor information. The input is the user's needs and sensor information, and the output is the generated suggestion message (e.g., "You're running low on milk. Would you like to order some?"). Specifically, the server creates a suggestion message based on the information that "milk is needed" and "stock is low."

[0151] Step 7: Sending a proposal message

[0152] The server generates a proposal message and sends it to the user's smartphone or smart speaker. The input is the proposal message generated in step 6, and the output is the message sent to the user's device. The specific operation is to send the message generated by the server to the user's smartphone as a notification.

[0153] Step 8: Receive user approval

[0154] The device collects the user's approval information and sends it to the server. The input is the user's approval (e.g., "Yes, please order"), and the output is the approval information sent to the server. Specifically, the user checks the notification on their smartphone and responds by saying "Yes, please order." The device collects this approval information and sends it to the server.

[0155] Step 9: Automatic Order Execution

[0156] The server uses the API of the e-commerce system to automatically place an order based on the user's approval. The input is the user's approval information and suggested product information, and the output is the completed order information. Specifically, the server uses the API of the online supermarket to automatically place an order for milk and send a confirmation email to the user.

[0157] Through each of the above processing steps, this system can automatically order the necessary items based on the user's conversation, significantly reducing the effort required for shopping.

[0158] (Application example 1)

[0159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] In everyday life, users often have to go through the trouble of ordering food or other items. Particularly in food delivery, users have to go through the trouble of opening an app to place an order and searching for specific food options. Therefore, to improve user convenience, there is a demand for a system that automatically orders the necessary items or meals by utilizing the user's everyday conversations and tweets.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0162] In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, means for analyzing the text data using natural language processing (NLP) to determine the user's needs, means for reinforcing the analysis results in conjunction with other sensor information, means for identifying necessary items and generating and sending suggested messages to the user, means for placing orders with e-commerce sites or online supermarkets based on the user's approval, and means for automatically placing orders using a food delivery API. This enables users to easily order items and meals through their everyday tweets and conversations.

[0163] "User" refers to any individual or corporation that uses this system.

[0164] A "conversation" refers to any or all of a user's spoken communication.

[0165] An "audio input device" is a device for collecting a user's voice, and includes, for example, a smartphone or smart glasses.

[0166] "Audio Data" refers to analog or digital audio signals collected by an audio input device.

[0167] "Text data" refers to text data converted from voice data using voice recognition technology.

[0168] "Natural language processing (NLP)" refers to technology that analyzes text data to understand its meaning and intent and identify user needs.

[0169] "Needs" refer to goods and services that users desire.

[0170] "Sensor Information" refers to information obtained from sensors that provide data related to a user or their environment.

[0171] "Suggestion Message" refers to a notification or message generated by the system to suggest desired goods or services to a user.

[0172] "Acceptance" refers to a positive response made by a user to a suggestion message.

[0173] "EC site" refers to a website for conducting electronic commerce.

[0174] An "online supermarket" refers to a website that sells groceries and daily necessities online.

[0175] A "food delivery API" refers to an interface that uses specific protocols and formats to exchange data between food delivery services and systems.

[0176] "Program" refers to a set of instructions that are executed on a computer system.

[0177] System configuration

[0178] A system for implementing the present invention includes the following components:

[0179] 1. Voice Input Device

[0180] A voice input device is a device equipped with a microphone for collecting a user's voice. Specifically, it includes a smartphone and smart glasses. This allows the device to collect a user's everyday conversations and tweets.

[0181] 2. Voice Recognition Technology

[0182] Voice data collected by the voice input device is sent to a cloud-based server, which is equipped with voice recognition technology (such as Google Cloud Speech-to-Text) and converts the voice data into text data.

[0183] 3. Natural Language Processing (NLP)

[0184] The server analyzes the converted text data using natural language processing (NLP) techniques (e.g., Transformer-based NLP models, BERT, GPT, etc.), and the analysis identifies the user's needs.

[0185] 4. Linking with sensor information

[0186] The server also integrates information provided by the refrigerator and indoor environmental sensors to augment the analysis results and more accurately identify the user's needs.

[0187] 5. Creating and sending a proposal message

[0188] The server generates a suggestion message based on the identified user needs, which is then sent as a notification to the user's smartphone, smart glasses, or other device, such as a message like, "Would you like to order curry?"

[0189] 6. User Authorization and Automated Orders

[0190] The user can approve the proposed message by voice or tap. This approval information is sent back to the server, and the order is automatically placed via the food delivery service's API (e.g., Uber Eats API).

[0191] Specific operation example

[0192] 1. Users tweet using voice

[0193] The user whispers into their smart glasses or smartphone, "I want to eat curry today."

[0194] 2. The audio data is sent to the server

[0195] The voice input device collects the user's voice and transmits the voice data to a server in real time.

[0196] 3. Generating text data using speech recognition

[0197] The server receives the voice data and converts it into text data such as "I'd like to eat curry today" using voice recognition technology such as Google Cloud Speech-to-Text.

[0198] 4. Identifying needs through natural language processing

[0199] The server analyzes this text data using natural language processing technology and identifies the need, such as "I need curry."

[0200] 5. Creating and Sending a Proposal Message

[0201] The server generates a suggestion message saying "Would you like to order curry?" and sends it to the user's smart device.

[0202] 6. User Authorization and Order Execution

[0203] When the user responds, "Yes, please order," the approval information is sent to the server, and the curry order is automatically placed through the food delivery API.

[0204] Example prompt

[0205] When a user says "I want to eat curry," generate a prompt that extracts the user's need to order "curry" as a result of natural language processing.

[0206] In this way, by using the system of the present invention, users can smoothly order meals from casual daily tweets, which is expected to significantly improve the convenience of users' lives.

[0207] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0208] Step 1:

[0209] The user speaks a specified phrase into the smart glasses or smartphone, which then collects the user's speech through the voice input device and generates audio data. The input is the user's voice, and the output is an audio file of the collected raw data.

[0210] Step 2:

[0211] The device collects audio data and sends it to the server in real time. The input is an audio file, and the output is the audio data sent to the server. The device uses the microphone and internet connection for audio input.

[0212] Step 3:

[0213] The server converts the received voice data into text using voice recognition technology such as Google Cloud Speech-to-Text. The input is voice data, and the converted text data is the output. This step converts the voice signal into text.

[0214] Step 4:

[0215] The server uses natural language processing (NLP) technology to analyze text data and identify user needs. The input is text data, and the analysis results, or needs information, are output. Here, a generative AI model (e.g., BERT or GPT) is used.

[0216] Step 5:

[0217] The server connects with sensor information obtained from the refrigerator and indoor environment sensors to augment the analysis results. The input is the needs information identified by NLP and sensor information, and the output is the augmented needs information. IoT devices are used to collect the sensor information.

[0218] Step 6:

[0219] The server generates a suggestion message based on the augmented needs information and sends it to the user's smart device. The input is the augmented needs information, and the output is the generated suggestion message. Here, an advanced text generation algorithm is used to generate the message.

[0220] Step 7:

[0221] The user receives the suggestion message, confirms it, and approves it by voice or tap. The input is the suggestion message, and the output is the user's approval information. The user's device plays a key role in this step.

[0222] Step 8:

[0223] The terminal sends the user's authorization information to the server. The input is the user's authorization information, and the output is the data sent to the server. The authorization information is again sent to the server via the Internet.

[0224] Step 9:

[0225] The server uses the food delivery service's API to automatically execute the order. The input is the user's approval information and needs information, and the output is a notification to the food delivery service that the order has been completed. The order is processed through communication using the API.

[0226] Examples of prompts used

[0227] When a user says "I want to eat curry," generate a prompt that extracts the user's need to order "curry" as a result of natural language processing.

[0228] In this way, the process from the user's voice input to the completion of the order is carried out smoothly, allowing the user to order the necessary goods or meals with less hassle.

[0229] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0230] The present invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. Specific embodiments of the system are described in detail below.

[0231] System configuration

[0232] 1. Voice Input Device

[0233] The terminal is equipped with a built-in voice input device equipped with a microphone, which constantly monitors the user's everyday conversations and tweets.

[0234] 2. Voice Recognition Technology

[0235] The server receives the voice data sent from the voice input device and converts the voice data into text data using voice recognition technology.

[0236] 3. Natural Language Processing (NLP)

[0237] The server passes the text data to a natural language processing (NLP) framework for analysis, which understands and identifies the user's needs and requirements.

[0238] 4. Linking with sensor information

[0239] The server will then integrate other sensor information (e.g., inventory sensors and temperature and humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs.

[0240] 5. Emotion Engine

[0241] The server is equipped with an emotion engine that analyzes emotions from the user's voice data, determining the user's emotions not only from the results of natural language processing but also from the tone, tempo, and other acoustic parameters of the voice.

[0242] 6. Proposal message generation and transmission

[0243] The server identifies the necessary items based on the analysis results and the emotion analysis results, and generates and sends a suggestion message to the user. The suggestion message is customized according to the user's emotion.

[0244] 7. User Authorization and Automated Orders

[0245] The terminal receives the user's approval and sends the approval information to the server, which then automatically places the order using the API of the online supermarket or e-commerce site based on the information.

[0246] Specific operation example

[0247] For example, consider the case where one day a user tweets, "I'm running out of milk..." and there is a certain amount of impatience or anxiety in the voice.

[0248] 1. Audio collection

[0249] A user says, "I'm running out of milk..."

[0250] The device collects this audio and sends it to the server.

[0251] 2. Voice Recognition

[0252] The server converts the audio data into text data.

[0253] The text data obtained is "We're running out of milk..."

[0254] 3. Natural Language Processing

[0255] The server uses natural language processing to analyze the text and extract the need, "I need milk."

[0256] 4. Emotion analysis

[0257] The server uses an emotion engine to recognize the user's impatience or anxiety as emotions from the voice data.

[0258] 5. Linking with sensor information

[0259] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[0260] 6. Proposal message generation and transmission

[0261] The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and, taking into account the user's impatience, also suggests an additional message saying, "It looks like you'll need to order soon."

[0262] The server sends a proposal message to the user's terminal.

[0263] 7. User Authorization and Order Execution

[0264] The user checks the notification on their smartphone and responds verbally, "Yes, please place the order."

[0265] The device sends this approval to the server.

[0266] The server automatically executes the milk order using the online supermarket's API.

[0267] In this way, by using the system of the present invention, necessary items can be ordered automatically and quickly and efficiently based on the user's tweets, and by using the emotion engine, it is possible to make suggestions according to the user's feelings and situation. As a result, a system that can respond more appropriately and quickly to user needs is realized.

[0268] The processing flow will be explained below.

[0269] Step 1:

[0270] The voice input device constantly monitors the user's daily conversations and tweets. When voice data is input, the device collects the voice data and starts recording.

[0271] Step 2:

[0272] The device sends the collected voice data to the server in real time, and the voice data is securely transmitted over the network.

[0273] Step 3:

[0274] The server converts the received voice data into text data using a speech recognition engine. A speech recognition algorithm is used in this process. For example, a voice saying "I'm running out of milk..." is converted into text data.

[0275] Step 4:

[0276] The server passes the text data to a natural language processing (NLP) framework for analysis. The analysis involves tokenizing the text, extracting nouns and verbs, and performing contextual analysis. This allows the need, "I need milk," to be extracted.

[0277] Step 5:

[0278] The server uses an emotion engine to analyze the user's emotions from the voice data. In this process, the server determines the user's emotions (e.g., impatience, joy, anxiety, etc.) from the tone, tempo, and other acoustic parameters of the voice. For example, if impatience is detected, the user is judged to be in a hurry.

[0279] Step 6:

[0280] The server determines the user's needs based on the analysis results and obtains sensor information from the refrigerator to confirm whether the identified items are truly necessary. For example, it checks whether there is a low amount of milk remaining using a camera or inventory sensor inside the refrigerator.

[0281] Step 7:

[0282] The server augments the analysis results with sensor information and ultimately identifies the item needed (in this case, milk).

[0283] Step 8:

[0284] The server generates a suggested message for the user, for example, "You're running low on milk. Would you like to order some?", and includes an additional message, "It looks like you need to order soon," taking into account the results of sentiment analysis (impatience).

[0285] Step 9:

[0286] The server sends the generated proposal message to the user's terminal, which notifies the user of the proposal message and waits for the user's response.

[0287] Step 10:

[0288] The user confirms the suggestion message and responds by voice, "Yes, I would like to order." Here, the user responds using a smartphone or voice input device.

[0289] Step 11:

[0290] The device sends the user's authorization information to the server, using secure communication methods.

[0291] Step 12:

[0292] The server receives the authorization information and automatically orders the specified item (milk) using the API of the online supermarket or e-commerce site. Specifically, it generates an order request and sends it to the API endpoint.

[0293] Step 13:

[0294] The server retrieves the order confirmation and notifies the user that the order has been completed, generating detailed notifications along the way, including success or failure status.

[0295] This series of steps not only enables a process for quickly and efficiently automatically ordering the necessary items based on the user's natural tweets and conversations, but also makes it possible to use an emotion engine to make optimal suggestions according to the situation.

[0296] Example 2

[0297] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0298] Conventional systems using voice input devices have difficulty accurately recognizing users' conversations and tweets and automatically ordering items. Furthermore, they are unable to make suggestions that take the user's emotions into account, making it difficult to place an order at the appropriate time. Furthermore, because they do not consider the user's level of urgency or emotions, the suggestions they make may not meet the user's needs.

[0299] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, means for analyzing the text data using natural language processing to determine the user's needs, means for reinforcing the analysis results in cooperation with other sensor information, means for determining the user's emotions using an emotion analysis engine, means for identifying required items and generating and transmitting a suggestion message taking the user's emotions into consideration, and means for placing an order with the e-commerce platform based on the user's approval. This makes it possible to appropriately identify needs from the user's conversations and tweets, generate a suggestion message taking the user's emotions and urgency into consideration, and automatically order items.

[0300] A "voice input device" is a hardware device that collects a user's speech and transmits it to a server as digital voice data.

[0301] "Voice data" is data that represents the user's speech in digital form.

[0302] "Character data" is data in text format converted from voice data.

[0303] "Natural language processing" is a technology that enables computers to understand and analyze human language.

[0304] "Sensor information" is environmental or condition data obtained from another device, such as temperature, humidity, or inventory status.

[0305] An "emotion analysis engine" is software that analyzes the tone and tempo of a user's speech and infers the user's emotions from that speech.

[0306] A "suggestion message" is a message generated by the system to suggest an item order to the user.

[0307] An "e-commerce platform" is an internet service that allows you to order and purchase products online.

[0308] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to automatically generate text or suggested messages.

[0309] "Server" means the central computing unit of the system, a computer used to analyze voice data, perform sentiment analysis and collaboration with other data, and execute orders for the necessary goods.

[0310] "User conversation" refers to the utterances and tweets that users make in their daily lives.

[0311] "Approval" is the act of the user agreeing to a proposed message from the system.

[0312] The present invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. Specific embodiments of the system are described in detail below.

[0313] System configuration

[0314] 1. Voice Input Device

[0315] The device is equipped with a built-in microphone and voice input device that constantly monitors the user's daily conversations and tweets. For example, a microphone with a noise-canceling function is used.

[0316] 2. Voice Recognition Technology

[0317] The server receives the voice data sent from the voice input device and converts it into text data using voice recognition technology, possibly using APIs such as Google Cloud Speech-to-Text.

[0318] 3. Natural Language Processing (NLP)

[0319] The server passes the text data to a natural language processing (NLP) framework for analysis, which then uses SpaCy and the Google Cloud Natural Language API to understand and identify the user's needs and requirements.

[0320] 4. Sentiment Analysis Engine

[0321] The server is equipped with an emotion analysis engine that analyzes emotions from the user's voice data. The emotion analysis engine determines the user's emotions not only from the results of natural language processing, but also from the tone and tempo of the voice and other acoustic parameters. IBM Watson (registered trademark) Tone Analyzer and other similar technologies are used.

[0322] 5. Linking with sensor information

[0323] The server will then link with other sensor information (e.g., inventory sensors and temperature / humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs. The smart refrigerator's built-in sensors and cameras will be used.

[0324] 6. Proposal message generation and transmission

[0325] The server identifies the necessary items based on the analysis results and the results of sentiment analysis, then generates a suggested message for the user and sends it to the device. The suggested message is customized according to the user's sentiment. A generative AI model (e.g., OpenAI® GPT-3®) is used to generate the suggested message.

[0326] 7. User Authorization and Automated Orders

[0327] The terminal receives the user's approval and sends the approval information to the server, which then automatically executes the order using the API of an e-commerce platform. Examples of such platforms include Amazon and Rakuten.

[0328] Specific operation example

[0329] For example, consider the case where one day a user tweets, "I'm running out of milk..." and there is a certain amount of impatience or anxiety in the voice.

[0330] 1. Audio collection

[0331] The user says, "I'm running out of milk..." The device collects this speech and sends it to the server.

[0332] 2. Voice Recognition

[0333] The server uses Google Cloud Speech-to-Text to convert the voice data into text data, which then becomes "We're running low on milk..."

[0334] 3. Natural Language Processing

[0335] The server uses SpaCy to analyze the text and extract the need "I need milk."

[0336] 4. Emotion analysis

[0337] The server uses IBM Watson Tone Analyzer to recognize the user's impatience or anxiety as emotions from the tone and tempo of the voice.

[0338] 5. Linking with sensor information

[0339] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[0340] 6. Proposal message generation and transmission

[0341] The server generates a suggested message saying, "You're running low on milk. Would you like to order some?" and, taking into account the user's impatience, suggests an additional message saying, "It looks like you need to order soon." These messages are generated using a generative AI model and sent to the device.

[0342] 7. User Authorization and Order Execution

[0343] The user sees the notification on their smartphone and responds verbally, "Yes, order." The device sends this approval to the server, which automatically fulfills the milk order using the e-commerce platform's API.

[0344] Prompt Sentence Examples

[0345] Here are some example prompts to input to the AI ​​model:

[0346] A user says, "Eggs are running low..." Use speech recognition technology to obtain the text data "Eggs are running low..." and use NLP to extract the need "I need eggs." Use an emotion engine to recognize the user's emotion (impatience or anxiety) from the tone of the voice, and combine this with egg inventory sensor information to generate a suggested message. The suggested message should include the basic message "We're running low on eggs. Would you like to order some?" as well as an additional message such as "It looks like you need to order soon." Finally, receive the user's approval and execute a flow that automatically orders eggs using the e-commerce platform's API.

[0347] As described above, by using the system of the present invention, necessary items can be automatically ordered quickly and efficiently based on the user's tweets, and by using the emotion engine, it is possible to make suggestions according to the user's feelings and situation. As a result, a system that can respond more appropriately and quickly to user needs can be realized.

[0348] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0349] Step 1: Audio Collection

[0350] One day, a user might say, "I'm running low on milk..." The microphone on the device collects the voice and transmits it to the server in real time. The noise-canceling function ensures clear voice data is transmitted. The input is the user's speech, and the output is digital voice data.

[0351] Step 2: Voice Recognition

[0352] The server converts the received voice data into text data using a speech recognition API such as Google Cloud Speech-to-Text. The input is voice data, and the output is the converted text data: "We're running out of milk..." This process converts the voice data into text format.

[0353] Step 3: Natural Language Processing (NLP)

[0354] The server passes the text data to a natural language processing framework (such as SpaCy or Google Cloud Natural Language API) for analysis. Through this analysis, the server extracts the user's need, "I need milk." The input is text data, and the output is the extracted need, "I need milk." The processing takes into account the context and identifies related keywords.

[0355] Step 4: Sentiment Analysis

[0356] The server passes the analyzed voice data to an emotion analysis engine (such as IBM Watson Tone Analyzer), which determines the user's emotion based on acoustic parameters such as tone, tempo, and volume. The input is the acoustic parameters of the voice data, and the output is a judgment result of the emotion, such as "impatience" or "anxiety."

[0357] Step 5: Linking with sensor information

[0358] The server acquires the inventory sensor information in the refrigerator and integrates it with the needs-based information. This information is collected from the inventory sensor in the refrigerator and retrieved using the smart refrigerator's API. The input is the needs and sensor information, and the output is the situation of "low milk stock."

[0359] Step 6: Generate and send a proposal message

[0360] The server generates a suggestion message based on the analysis results and sentiment analysis results. Using a generative AI model (e.g., OpenAI GPT-3), it generates the message "You're running low on milk. Would you like to order some?", plus an additional message such as "It looks like you need to order soon." The input is the need, sentiment analysis results, and sensor information, and the output is the generated suggestion message. The server sends the generated message to the device.

[0361] Step 7: User approval and order execution

[0362] The user confirms the notification on their smartphone and responds verbally with "Yes, please place the order." The device collects this approval and sends it to the server, which then automatically fulfills the milk order using the API of an e-commerce platform (e.g., Amazon or Rakuten Ichiba). The input is the user's approval, and the output is a confirmation message that the order has been fulfilled.

[0363] Through the above processing steps, the system can properly identify needs from the user's speech, generate and send a suggested message that takes into account the user's emotions and situation, and automatically order the item.

[0364] (Application example 2)

[0365] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0366] In modern society, users often find it difficult to order necessary items in a timely manner due to their busy daily lives. Furthermore, there is a lack of systems that provide appropriate suggestions based on the user's emotions and urgency. Conventional systems have difficulty accurately analyzing the user's emotions and making suggestions that reflect those emotions. Therefore, there is a need not only for systems that can automatically order necessary items based on the user's conversations and tweets, but also for systems that can customize the suggestions based on the user's emotions. Furthermore, there is a need to apply generative AI models to enhance the suggestions.

[0367] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0368] In this invention, the server includes: means for collecting user conversations using a voice input device; means for converting the collected voice data into text data; means for analyzing the text data using natural language processing (NLP) to determine the user's needs; means for reinforcing the analysis results by linking with other sensor information; means for identifying necessary items and generating and sending a suggested message to the user; means for placing an order on an e-commerce site or online sales platform based on the user's approval; means including an emotion analysis engine for analyzing the user's emotions from the conversation and reflecting the results in the suggested message; and means for inputting the suggested message as a prompt sentence into a generative AI model to reinforce the suggested content. This makes it possible to provide suggestions based on the user's needs and emotions, enabling the user to quickly and appropriately order the necessary items.

[0369] A "voice input device" is a device such as a microphone for collecting a user's conversations and tweets.

[0370] "Voice data" refers to data in which the user's voice is recorded in digital format.

[0371] "Character data" is data in text format obtained by analyzing voice data.

[0372] "Natural Language Processing (NLP)" is a technology for analyzing human language and understanding its meaning and intent.

[0373] "Sensor information" is, for example, environmental data obtained from inventory sensors and temperature / humidity sensors inside a refrigerator.

[0374] A "suggestion message" is a message containing a suggestion for ordering an item that is generated based on the user's needs and emotions.

[0375] An "emotion analysis engine" is a technology that analyzes a user's emotions from voice data and obtains the results.

[0376] A "generative AI model" is an artificial intelligence model that performs generative processing using large amounts of training data, and is used to enhance the content of suggestions.

[0377] A "prompt sentence" is an input sentence that gives instructions to a generative AI model.

[0378] An "ecommerce site or online sales platform" is a website or application that sells goods or services over the internet.

[0379] An "order fulfillment means" is a system or process for automatically ordering items based on user approval.

[0380] This invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. This system is realized using the following hardware and software.

[0381] System configuration

[0382] 1. Voice Input Device

[0383] The terminal is equipped with a built-in microphone and voice input device that constantly monitors the user's everyday conversations and tweets. This voice data is sent to a server via the Internet.

[0384] 2. Server Roles

[0385] The server is equipped with multiple software modules for performing the following processes:

[0386] Voice Recognition

[0387] We use voice recognition software (e.g., Google Cloud Speech-to-Text) to convert collected voice data into text data.

[0388] Natural Language Processing (NLP)

[0389] Use a natural language processing (NLP) framework (e.g., spaCy, NLTK) to analyze text data and determine user needs.

[0390] Sentiment Analysis Engine

[0391] An emotion analysis engine (e.g., IBM Watson) is used to analyze user emotions from voice data and obtain the results.

[0392] Linking with sensor information

[0393] It works in conjunction with other sensor information (e.g., inventory sensors and temperature / humidity sensors inside the refrigerator) to reinforce the analysis results.

[0394] Using generative AI models

[0395] The suggested message is input as a prompt into the generative AI model to reinforce the suggestion.

[0396] 3. Creating and sending a proposal message

[0397] The server generates a suggestion message based on the analysis results and sentiment analysis results and sends it to the user's device. The generated suggestion message is enhanced using the generative AI model.

[0398] 4. User Authorization and Automated Orders

[0399] The terminal receives the user's approval and sends the approval information to the server, which then automatically executes the order using the API of the e-commerce site or online sales platform.

[0400] Specific examples

[0401] For example, consider the case where the user says, "I'm a little tired today... I want to eat some delicious pizza." The processing flow in this case is as follows:

[0402] 1. User says: "I'm feeling a bit tired today... I want to eat some good pizza."

[0403] 2. Collecting voice data: The device's microphone collects voice and sends it to the server.

[0404] 3. Speech Recognition: Speech recognition software converts voice data into text data.

[0405] 4. Natural language processing: A natural language processing framework analyzes the text data and extracts the need, "I want to eat pizza."

[0406] 5. Sentiment analysis: The sentiment analysis engine recognizes the emotion "tired" from the user's speech.

[0407] 6. Generate suggestion message: The server generates a suggestion message such as, "You seem tired and want a nice pizza. Would you like us to order a pizza for delivery?" and sends it to the user's device.

[0408] 7. Use of generative AI model: The proposed message is input to the generative AI model as a prompt sentence and further enhanced. Specific examples of prompt sentences are as follows:

[0409] Prompt Sentence Examples

[0410] "A user said, 'I'm a little tired today... I want to eat some good pizza.' He seems tired. Please create a message suggesting delivery."

[0411] 8. User approval and order fulfillment: Once the user approves the proposed message, the server automatically fulfills the pizza order using the API of the e-commerce site or online sales platform.

[0412] As described above, the embodiment of the present invention has been specifically described. This system provides suggestions based on the user's needs and emotions, and enables the user to order goods quickly and appropriately.

[0413] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0414] Step 1:

[0415] (Audio collection)

[0416] The device uses a microphone to collect the user's speech. For example, the user might say, "I'm a little tired today... I want to eat some delicious pizza." The collected voice data is sent to a server via the Internet.

[0417] Input: User's voice

[0418] Output: Audio data

[0419] Step 2:

[0420] (Voice Recognition)

[0421] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the collected voice data into text data.

[0422] Input: Audio data

[0423] Output: Character data

[0424] Step 3:

[0425] (Natural Language Processing (NLP))

[0426] The server uses a natural language processing framework (e.g., spaCy, NLTK) to analyze the text data and determine the user's needs. Specifically, it extracts the need "I want pizza."

[0427] Input: Character data

[0428] Output: Needs data "I want to eat pizza"

[0429] Step 4:

[0430] (emotional analysis)

[0431] The server uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's emotions from the voice data. Specifically, it recognizes the emotion "tired" from the way the user speaks and the tone of their voice.

[0432] Input: Audio data

[0433] Output: Emotion data of "tired"

[0434] Step 5:

[0435] (Sensor information integration)

[0436] The server retrieves data from the refrigerator's inventory sensor to check whether pizza ingredients are available, and integrates the sensor information into the analysis results to reinforce the recommendation message.

[0437] Input: Refrigerator inventory information

[0438] Output: Consolidated data

[0439] Step 6:

[0440] (Generating a proposal message)

[0441] The server generates a suggestion message based on the need and emotion of "I want pizza, but I'm tired." Specifically, it creates a message that reads, "It seems you want to eat delicious pizza, but you're tired. Would you like us to order pizza for delivery?"

[0442] Input: Needs data such as "I want to eat pizza," emotion data such as "I'm tired," and sensor information

[0443] Output: Proposal message

[0444] Step 7:

[0445] (Using generative AI models)

[0446] The server provides a prompt to the generative AI model to enhance the suggestion message. An example of a prompt is, "The user said, 'I'm a little tired today... I want to eat some good pizza.' He seems tired. Please create a suggestion message that encourages delivery." An enhanced suggestion message is generated.

[0447] Input: Proposal message, prompt text

[0448] Output: Enhanced suggestion message

[0449] Step 8:

[0450] (Send a proposal message)

[0451] The server sends the enhanced proposal message to the terminal, and the user's terminal displays the proposal message for the user to confirm.

[0452] Input: Enhanced suggestion message

[0453] Output: The suggestion message that is displayed on the user's terminal.

[0454] Step 9:

[0455] (User approval and automatic ordering)

[0456] The user approves the proposed message (for example, by saying "Yes, please place the order"), and the device sends this approval to the server, which then executes the order using the API of the e-commerce site or online sales platform.

[0457] Input: User approval

[0458] Output: Order execution

[0459] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0460] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0461] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0462] [Second embodiment]

[0463] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0464] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0465] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0466] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0467] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0468] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0469] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0470] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0471] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0472] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0473] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0474] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0475] This invention is a system that utilizes user conversations and tweets to automatically order necessary items. This system collects and analyzes the user's voice, identifies the necessary items, makes ordering suggestions to the user, and, if the user approves, automatically places the order with an online supermarket or e-commerce site.

[0476] System configuration

[0477] 1. Voice Input Device

[0478] The terminal is equipped with a built-in voice input device equipped with a microphone, which constantly monitors the user's everyday conversations and tweets.

[0479] 2. Voice Recognition Technology

[0480] The server receives the voice data sent from the voice input device and converts the voice data into text data using voice recognition technology.

[0481] 3. Natural Language Processing (NLP)

[0482] The server analyzes the text data using natural language processing (NLP) to understand and identify the user's needs and requirements.

[0483] 4. Linking with sensor information

[0484] The server will then integrate other sensor information (e.g., inventory sensors and temperature and humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs.

[0485] 5. Creating and sending a proposal message

[0486] The server identifies the items the user needs based on the analysis results, generates a suggestion message for the user, and sends it to the user's device. The suggestion message is then delivered to the user via a smartphone notification or voice command.

[0487] 6. User Authorization and Automated Orders

[0488] The terminal receives the user's approval and sends that approval information to the server, which then uses the API of the online supermarket or e-commerce site to automatically place the order based on that information.

[0489] Specific operation example

[0490] For example, consider the case where a user tweets one day, "I'm running out of milk..."

[0491] 1. Audio collection

[0492] A user says, "I'm running out of milk..."

[0493] The device collects this audio and sends it to the server.

[0494] 2. Voice Recognition

[0495] The server converts the audio data into text data.

[0496] The text data obtained is "We're running out of milk..."

[0497] 3. Natural Language Processing

[0498] The server uses natural language processing to analyze the text and extract the need, "I need milk."

[0499] 4. Linking with sensor information

[0500] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[0501] 5. Creating and sending a proposal message

[0502] The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and sends it to the user's smartphone.

[0503] 6. User Authorization and Order Execution

[0504] The user checks the notification on their smartphone and responds verbally, "Yes, please place the order."

[0505] The device sends this approval to the server.

[0506] The server automatically executes the milk order using the online supermarket's API.

[0507] In this way, by using the system of the present invention, users can automatically order the items they need based on casual daily tweets, significantly reducing the effort required for shopping.

[0508] The processing flow will be explained below.

[0509] Step 1:

[0510] The voice input device constantly monitors the user's daily conversations and tweets. When voice data is input, the device collects the voice data and starts recording.

[0511] Step 2:

[0512] The device sends the collected voice data to the server in real time, and the voice data is securely transmitted over the network.

[0513] Step 3:

[0514] The server converts the received voice data into text data using a voice recognition engine, which uses a voice recognition algorithm.

[0515] Step 4:

[0516] The server passes the text data to a natural language processing (NLP) framework for analysis, which involves tokenizing the text into words, extracting nouns and verbs, and performing contextual analysis.

[0517] Step 5:

[0518] The server determines the user's intentions and needs from the analysis results. For example, if someone tweets "I'm running out of milk...", it will recognize that "I need milk."

[0519] Step 6:

[0520] The server checks the user's smart device and IoT sensor information, for example, checking the refrigerator's internal camera and inventory sensor to see if there is a low level of milk.

[0521] Step 7:

[0522] The server augments the analysis with sensor information and ultimately identifies the items needed. This specific step verifies that the sensor readings match expectations.

[0523] Step 8:

[0524] The server identifies the items needed and generates a suggestion message based on that, for example, "You're running low on milk. Would you like to order some?"

[0525] Step 9:

[0526] The server sends the generated proposal message to the user's terminal via a transmission protocol. The terminal notifies the user of the proposal message and waits for a response from the user.

[0527] Step 10:

[0528] The user reviews the suggestion message and takes an action to accept or reject it, for example, by saying "Yes, please place the order."

[0529] Step 11:

[0530] The device sends the user's authorization information to the server, using secure communication methods.

[0531] Step 12:

[0532] The server receives the authorization information and automatically orders the identified items using the API of the e-commerce site or online supermarket. Specifically, it generates an order request and sends it to the API endpoint.

[0533] Step 13:

[0534] The server retrieves the order confirmation and notifies the user that the order has been completed, generating detailed notifications along the way, including success or failure status.

[0535] This series of steps realizes a process for quickly and efficiently automatically ordering the necessary items from the user's spontaneous tweets.

[0536] Example 1

[0537] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0538] Conventional e-commerce systems require users to manually order the items they need each time, which is time-consuming and often leads to users forgetting to order. It is also difficult to grasp the user's situation and needs in real time and automate appropriate suggestions and orders based on that information. There is a need for a system that can solve these problems and enable users to purchase goods more conveniently and efficiently.

[0539] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0540] In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, and means for analyzing the text data using natural language processing (NLP) to determine the user's needs. This enables the server to automatically determine the items needed from the user's tweets and conversations, generate and send suggestion messages, and enable efficient purchasing.

[0541] "User" refers to an individual or business entity that uses the system.

[0542] "Audio input device" refers to a hardware device with a built-in microphone that collects a user's voice.

[0543] "Voice Data" refers to digitally recorded data of a user's voice collected by a voice input device.

[0544] "Character data" refers to text-format data converted from voice data using voice recognition technology.

[0545] "Natural Language Processing (NLP)" refers to the technology of analyzing text data to understand and identify user needs and requirements.

[0546] "Sensor information" refers to data obtained from IoT devices such as inventory sensors and temperature / humidity sensors inside the refrigerator.

[0547] A "suggestion message" refers to a message that is generated based on the analysis results and that suggests to the user to purchase an item.

[0548] "E-commerce system" refers to an online platform that enables the purchase of goods and services via the Internet.

[0549] "Dynamic pricing" refers to a pricing method in which prices fluctuate according to supply and demand.

[0550] "Approval information" refers to data provided by a user to indicate approval of a proposed message.

[0551] This invention is a system that automatically orders necessary items by utilizing user conversations and tweets. This system consists of a voice input device, i.e., a terminal equipped with a microphone for collecting the user's voice, and a server for processing this voice.

[0552] The device is equipped with a voice input device such as a smart speaker, which constantly collects the user's everyday conversations and tweets. For example, if a user tweets, "I'm running out of milk...", the voice data is encrypted and sent to a server in real time. The server then converts the voice data into text using voice recognition technologies such as the Google Speech-to-Text API or Azure Speech-to-Text.

[0553] The server then analyzes this text data using natural language processing (NLP) techniques such as BERT. This analysis determines the user's specific needs and requests, such as "I need milk." The server then checks the refrigerator's inventory, temperature, and humidity sensors to reinforce the analysis. Based on this information, the server determines whether the item the user needs is actually in short supply.

[0554] Once the server identifies the need, it generates a suggestion message saying, "You're low on milk. Would you like to order some?" This message is sent to the user via their smartphone or smart speaker. When the user responds to this suggestion with "Yes, please order," the device collects the voice and sends it back to the server.

[0555] Once the server confirms the user's approval, it uses the API of the e-commerce system to automatically place the order. For example, it automates the ordering process at online supermarkets and e-commerce sites, and payments are made using PayPal or credit card information. Once the order is complete, a confirmation email is sent to the user.

[0556] Specific examples

[0557] Example 1: When you are running out of milk

[0558] 1. A user tweets, "I'm running out of milk..."

[0559] 2. The device collects the audio and sends it to the server.

[0560] 3. The server uses the Google Speech-to-Text API to convert the audio data into text data.

[0561] 4. The server uses BERT to extract the need "I need milk."

[0562] 5. The server retrieves data from the refrigerator's inventory sensor and checks for low inventory.

[0563] 6. The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and sends it to the smartphone.

[0564] 7. The user responds, "Yes, place the order."

[0565] 8. The device sends the authorization information to the server.

[0566] 9. The server automatically orders milk using the API of the e-commerce system.

[0567] This system allows users to automatically order the items they need based on casual daily tweets, significantly reducing the effort required for shopping.

[0568] Examples of prompt statements

[0569] "Please tell me how to collect tweets about how I'm running out of milk, determine their needs using natural language processing, and send them suggestion messages."

[0570] "Please tell me the specific steps of a system that uses voice recognition technology to identify the items needed from the user's voice and automatically places an order with an online supermarket."

[0571] As described above, the present invention utilizes users' conversations and tweets, and makes use of voice input devices and advanced analysis technology to make users' lives more convenient.

[0572] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0573] Step 1: Collect audio input

[0574] The device uses a built-in microphone to constantly collect the user's everyday conversations and tweets. The input is the user's voice, which is converted into digital voice data. Specifically, when a user tweets, "I'm running out of milk...", the device collects the voice in real time. The output is the collected voice data.

[0575] Step 2: Sending audio data

[0576] The voice data collected by the terminal is encrypted and sent to the server via the Internet. The input is the digital voice data collected in step 1, and the output is the voice data sent to the server. Specifically, the terminal converts the collected voice data into packets and sends them to the server.

[0577] Step 3: Voice Recognition

[0578] The server uses speech recognition technology such as Google Speech-to-Text API or Azure Speech-to-Text to convert the voice data into text data. The input is the voice data sent in step 2, and the output is text data. Specifically, the server analyzes the voice data and generates text data such as "I'm running low on milk..."

[0579] Step 4: Natural Language Processing (NLP)

[0580] The server uses natural language processing (NLP) techniques such as BERT to analyze the text data and identify the user's needs. The input is the text data generated in step 3, and the output is the user's identified need (e.g., "I need milk"). Specifically, the server analyzes the text data and extracts the need "I need milk" from "I'm running low on milk..."

[0581] Step 5: Acquire sensor information

[0582] The server obtains data from the inventory sensor and other sensors in the refrigerator to augment the analysis results. The input is data from the inventory sensor and temperature / humidity sensors in the refrigerator, and the output is the confirmed inventory status (e.g., "Milk is low in stock"). Specifically, the server obtains information that "Milk is low in stock" from the inventory sensor in the refrigerator.

[0583] Step 6: Proposal message generation

[0584] The server generates a suggestion message for the user based on the analysis results and sensor information. The input is the user's needs and sensor information, and the output is the generated suggestion message (e.g., "You're running low on milk. Would you like to order some?"). Specifically, the server creates a suggestion message based on the information that "milk is needed" and "stock is low."

[0585] Step 7: Sending a proposal message

[0586] The server generates a proposal message and sends it to the user's smartphone or smart speaker. The input is the proposal message generated in step 6, and the output is the message sent to the user's device. The specific operation is to send the message generated by the server to the user's smartphone as a notification.

[0587] Step 8: Receive user approval

[0588] The device collects the user's approval information and sends it to the server. The input is the user's approval (e.g., "Yes, please order"), and the output is the approval information sent to the server. Specifically, the user checks the notification on their smartphone and responds by saying "Yes, please order." The device collects this approval information and sends it to the server.

[0589] Step 9: Automatic Order Execution

[0590] The server uses the API of the e-commerce system to automatically place an order based on the user's approval. The input is the user's approval information and suggested product information, and the output is the completed order information. Specifically, the server uses the API of the online supermarket to automatically place an order for milk and send a confirmation email to the user.

[0591] Through each of the above processing steps, this system can automatically order the necessary items based on the user's conversation, significantly reducing the effort required for shopping.

[0592] (Application example 1)

[0593] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0594] In everyday life, users often have to go through the trouble of ordering food or other items. Particularly in food delivery, users have to go through the trouble of opening an app to place an order and searching for specific food options. Therefore, to improve user convenience, there is a demand for a system that automatically orders the necessary items or meals by utilizing the user's everyday conversations and tweets.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0596] In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, means for analyzing the text data using natural language processing (NLP) to determine the user's needs, means for reinforcing the analysis results in conjunction with other sensor information, means for identifying necessary items and generating and sending suggested messages to the user, means for placing orders with e-commerce sites or online supermarkets based on the user's approval, and means for automatically placing orders using a food delivery API. This enables users to easily order items and meals through their everyday tweets and conversations.

[0597] "User" refers to any individual or corporation that uses this system.

[0598] A "conversation" refers to any or all of a user's spoken communication.

[0599] An "audio input device" is a device for collecting a user's voice, and includes, for example, a smartphone or smart glasses.

[0600] "Audio Data" refers to analog or digital audio signals collected by an audio input device.

[0601] "Text data" refers to text data converted from voice data using voice recognition technology.

[0602] "Natural language processing (NLP)" refers to technology that analyzes text data to understand its meaning and intent and identify user needs.

[0603] "Needs" refer to goods and services that users desire.

[0604] "Sensor Information" refers to information obtained from sensors that provide data related to a user or their environment.

[0605] "Suggestion Message" refers to a notification or message generated by the system to suggest desired goods or services to a user.

[0606] "Acceptance" refers to a positive response made by a user to a suggestion message.

[0607] "EC site" refers to a website for conducting electronic commerce.

[0608] An "online supermarket" refers to a website that sells groceries and daily necessities online.

[0609] A "food delivery API" refers to an interface that uses specific protocols and formats to exchange data between food delivery services and systems.

[0610] "Program" refers to a set of instructions that are executed on a computer system.

[0611] System configuration

[0612] A system for implementing the present invention includes the following components:

[0613] 1. Voice Input Device

[0614] A voice input device is a device equipped with a microphone for collecting a user's voice. Specifically, it includes a smartphone and smart glasses. This allows the device to collect a user's everyday conversations and tweets.

[0615] 2. Voice Recognition Technology

[0616] Voice data collected by the voice input device is sent to a cloud-based server, which is equipped with voice recognition technology (such as Google Cloud Speech-to-Text) and converts the voice data into text data.

[0617] 3. Natural Language Processing (NLP)

[0618] The server analyzes the converted text data using natural language processing (NLP) techniques (e.g., Transformer-based NLP models, BERT, GPT, etc.), and the analysis identifies the user's needs.

[0619] 4. Linking with sensor information

[0620] The server also integrates information provided by the refrigerator and indoor environmental sensors to augment the analysis results and more accurately identify the user's needs.

[0621] 5. Creating and sending a proposal message

[0622] The server generates a suggestion message based on the identified user needs, which is then sent as a notification to the user's smartphone, smart glasses, or other device, such as a message like, "Would you like to order curry?"

[0623] 6. User Authorization and Automated Orders

[0624] The user can approve the proposed message by voice or tap. This approval information is sent back to the server, and the order is automatically placed via the food delivery service's API (e.g., Uber Eats API).

[0625] Specific operation example

[0626] 1. Users tweet using voice

[0627] The user whispers into their smart glasses or smartphone, "I want to eat curry today."

[0628] 2. The audio data is sent to the server

[0629] The voice input device collects the user's voice and transmits the voice data to a server in real time.

[0630] 3. Generating text data using speech recognition

[0631] The server receives the voice data and converts it into text data such as "I'd like to eat curry today" using voice recognition technology such as Google Cloud Speech-to-Text.

[0632] 4. Identifying needs through natural language processing

[0633] The server analyzes this text data using natural language processing technology and identifies the need, such as "I need curry."

[0634] 5. Creating and Sending a Proposal Message

[0635] The server generates a suggestion message saying "Would you like to order curry?" and sends it to the user's smart device.

[0636] 6. User Authorization and Order Execution

[0637] When the user responds, "Yes, please order," the approval information is sent to the server, and the curry order is automatically placed through the food delivery API.

[0638] Example prompt

[0639] When a user says "I want to eat curry," generate a prompt that extracts the user's need to order "curry" as a result of natural language processing.

[0640] In this way, by using the system of the present invention, users can smoothly order meals from casual daily tweets, which is expected to significantly improve the convenience of users' lives.

[0641] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0642] Step 1:

[0643] The user speaks a specified phrase into the smart glasses or smartphone, which then collects the user's speech through the voice input device and generates audio data. The input is the user's voice, and the output is an audio file of the collected raw data.

[0644] Step 2:

[0645] The device collects audio data and sends it to the server in real time. The input is an audio file, and the output is the audio data sent to the server. The device uses the microphone and internet connection for audio input.

[0646] Step 3:

[0647] The server converts the received voice data into text using voice recognition technology such as Google Cloud Speech-to-Text. The input is voice data, and the converted text data is the output. This step converts the voice signal into text.

[0648] Step 4:

[0649] The server uses natural language processing (NLP) technology to analyze text data and identify user needs. The input is text data, and the analysis results, or needs information, are output. Here, a generative AI model (e.g., BERT or GPT) is used.

[0650] Step 5:

[0651] The server connects with sensor information obtained from the refrigerator and indoor environment sensors to augment the analysis results. The input is the needs information identified by NLP and sensor information, and the output is the augmented needs information. IoT devices are used to collect the sensor information.

[0652] Step 6:

[0653] The server generates a suggestion message based on the augmented needs information and sends it to the user's smart device. The input is the augmented needs information, and the output is the generated suggestion message. Here, an advanced text generation algorithm is used to generate the message.

[0654] Step 7:

[0655] The user receives the suggestion message, confirms it, and approves it by voice or tap. The input is the suggestion message, and the output is the user's approval information. The user's device plays a key role in this step.

[0656] Step 8:

[0657] The terminal sends the user's authorization information to the server. The input is the user's authorization information, and the output is the data sent to the server. The authorization information is again sent to the server via the Internet.

[0658] Step 9:

[0659] The server uses the food delivery service's API to automatically execute the order. The input is the user's approval information and needs information, and the output is a notification to the food delivery service that the order has been completed. The order is processed through communication using the API.

[0660] Examples of prompts used

[0661] When a user says "I want to eat curry," generate a prompt that extracts the user's need to order "curry" as a result of natural language processing.

[0662] In this way, the process from the user's voice input to the completion of the order is carried out smoothly, allowing the user to order the necessary goods or meals with less hassle.

[0663] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0664] The present invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. Specific embodiments of the system are described in detail below.

[0665] System configuration

[0666] 1. Voice Input Device

[0667] The terminal is equipped with a built-in voice input device equipped with a microphone, which constantly monitors the user's everyday conversations and tweets.

[0668] 2. Voice Recognition Technology

[0669] The server receives the voice data sent from the voice input device and converts the voice data into text data using voice recognition technology.

[0670] 3. Natural Language Processing (NLP)

[0671] The server passes the text data to a natural language processing (NLP) framework for analysis, which understands and identifies the user's needs and requirements.

[0672] 4. Linking with sensor information

[0673] The server will then integrate other sensor information (e.g., inventory sensors and temperature and humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs.

[0674] 5. Emotion Engine

[0675] The server is equipped with an emotion engine that analyzes emotions from the user's voice data, determining the user's emotions not only from the results of natural language processing but also from the tone, tempo, and other acoustic parameters of the voice.

[0676] 6. Proposal message generation and transmission

[0677] The server identifies the necessary items based on the analysis results and the emotion analysis results, and generates and sends a suggestion message to the user. The suggestion message is customized according to the user's emotion.

[0678] 7. User Authorization and Automated Orders

[0679] The terminal receives the user's approval and sends the approval information to the server, which then automatically places the order using the API of the online supermarket or e-commerce site based on the information.

[0680] Specific operation example

[0681] For example, consider the case where one day a user tweets, "I'm running out of milk..." and there is a certain amount of impatience or anxiety in the voice.

[0682] 1. Audio collection

[0683] A user says, "I'm running out of milk..."

[0684] The device collects this audio and sends it to the server.

[0685] 2. Voice Recognition

[0686] The server converts the audio data into text data.

[0687] The text data obtained is "We're running out of milk..."

[0688] 3. Natural Language Processing

[0689] The server uses natural language processing to analyze the text and extract the need, "I need milk."

[0690] 4. Emotion analysis

[0691] The server uses an emotion engine to recognize the user's impatience or anxiety as emotions from the voice data.

[0692] 5. Linking with sensor information

[0693] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[0694] 6. Proposal message generation and transmission

[0695] The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and, taking into account the user's impatience, also suggests an additional message saying, "It looks like you'll need to order soon."

[0696] The server sends a proposal message to the user's terminal.

[0697] 7. User Authorization and Order Execution

[0698] The user checks the notification on their smartphone and responds verbally, "Yes, please place the order."

[0699] The device sends this approval to the server.

[0700] The server automatically executes the milk order using the online supermarket's API.

[0701] In this way, by using the system of the present invention, necessary items can be ordered automatically and quickly and efficiently based on the user's tweets, and by using the emotion engine, it is possible to make suggestions according to the user's feelings and situation. As a result, a system that can respond more appropriately and quickly to user needs is realized.

[0702] The processing flow will be explained below.

[0703] Step 1:

[0704] The voice input device constantly monitors the user's daily conversations and tweets. When voice data is input, the device collects the voice data and starts recording.

[0705] Step 2:

[0706] The device sends the collected voice data to the server in real time, and the voice data is securely transmitted over the network.

[0707] Step 3:

[0708] The server converts the received voice data into text data using a speech recognition engine. A speech recognition algorithm is used in this process. For example, a voice saying "I'm running out of milk..." is converted into text data.

[0709] Step 4:

[0710] The server passes the text data to a natural language processing (NLP) framework for analysis. The analysis involves tokenizing the text, extracting nouns and verbs, and performing contextual analysis. This allows the need, "I need milk," to be extracted.

[0711] Step 5:

[0712] The server uses an emotion engine to analyze the user's emotions from the voice data. In this process, the server determines the user's emotions (e.g., impatience, joy, anxiety, etc.) from the tone, tempo, and other acoustic parameters of the voice. For example, if impatience is detected, the user is judged to be in a hurry.

[0713] Step 6:

[0714] The server determines the user's needs based on the analysis results and obtains sensor information from the refrigerator to confirm whether the identified items are truly necessary. For example, it checks whether there is a low amount of milk remaining using a camera or inventory sensor inside the refrigerator.

[0715] Step 7:

[0716] The server augments the analysis results with sensor information and ultimately identifies the item needed (in this case, milk).

[0717] Step 8:

[0718] The server generates a suggested message for the user, for example, "You're running low on milk. Would you like to order some?", and includes an additional message, "It looks like you need to order soon," taking into account the results of sentiment analysis (impatience).

[0719] Step 9:

[0720] The server sends the generated proposal message to the user's terminal, which notifies the user of the proposal message and waits for the user's response.

[0721] Step 10:

[0722] The user confirms the suggestion message and responds by voice, "Yes, I would like to order." Here, the user responds using a smartphone or voice input device.

[0723] Step 11:

[0724] The device sends the user's authorization information to the server, using secure communication methods.

[0725] Step 12:

[0726] The server receives the authorization information and automatically orders the specified item (milk) using the API of the online supermarket or e-commerce site. Specifically, it generates an order request and sends it to the API endpoint.

[0727] Step 13:

[0728] The server retrieves the order confirmation and notifies the user that the order has been completed, generating detailed notifications along the way, including success or failure status.

[0729] This series of steps not only enables a process for quickly and efficiently automatically ordering the necessary items based on the user's natural tweets and conversations, but also makes it possible to use an emotion engine to make optimal suggestions according to the situation.

[0730] Example 2

[0731] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0732] Conventional systems using voice input devices have difficulty accurately recognizing users' conversations and tweets and automatically ordering items. Furthermore, they are unable to make suggestions that take the user's emotions into account, making it difficult to place an order at the appropriate time. Furthermore, because they do not consider the user's level of urgency or emotions, the suggestions they make may not meet the user's needs.

[0733] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, means for analyzing the text data using natural language processing to determine the user's needs, means for reinforcing the analysis results in cooperation with other sensor information, means for determining the user's emotions using an emotion analysis engine, means for identifying required items and generating and transmitting a suggestion message taking the user's emotions into consideration, and means for placing an order with the e-commerce platform based on the user's approval. This makes it possible to appropriately identify needs from the user's conversations and tweets, generate a suggestion message taking the user's emotions and urgency into consideration, and automatically order items.

[0734] A "voice input device" is a hardware device that collects a user's speech and transmits it to a server as digital voice data.

[0735] "Voice data" is data that represents the user's speech in digital form.

[0736] "Character data" is data in text format converted from voice data.

[0737] "Natural language processing" is a technology that enables computers to understand and analyze human language.

[0738] "Sensor information" is environmental or condition data obtained from another device, such as temperature, humidity, or inventory status.

[0739] An "emotion analysis engine" is software that analyzes the tone and tempo of a user's speech and infers the user's emotions from that speech.

[0740] A "suggestion message" is a message generated by the system to suggest an item order to the user.

[0741] An "e-commerce platform" is an internet service that allows you to order and purchase products online.

[0742] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to automatically generate text or suggested messages.

[0743] "Server" means the central computing unit of the system, a computer used to analyze voice data, perform sentiment analysis and collaboration with other data, and execute orders for the necessary goods.

[0744] "User conversation" refers to the utterances and tweets that users make in their daily lives.

[0745] "Approval" is the act of the user agreeing to a proposed message from the system.

[0746] The present invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. Specific embodiments of the system are described in detail below.

[0747] System configuration

[0748] 1. Voice Input Device

[0749] The device is equipped with a built-in microphone and voice input device that constantly monitors the user's daily conversations and tweets. For example, a microphone with a noise-canceling function is used.

[0750] 2. Voice Recognition Technology

[0751] The server receives the voice data sent from the voice input device and converts it into text data using voice recognition technology, possibly using APIs such as Google Cloud Speech-to-Text.

[0752] 3. Natural Language Processing (NLP)

[0753] The server passes the text data to a natural language processing (NLP) framework for analysis, which then uses SpaCy and the Google Cloud Natural Language API to understand and identify the user's needs and requirements.

[0754] 4. Sentiment Analysis Engine

[0755] The server is equipped with an emotion analysis engine that analyzes emotions from the user's voice data. The emotion analysis engine determines the user's emotions not only from the results of natural language processing, but also from the tone, tempo, and other acoustic parameters of the voice. It uses a system such as IBM Watson Tone Analyzer.

[0756] 5. Linking with sensor information

[0757] The server will then link with other sensor information (e.g., inventory sensors and temperature / humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs. The smart refrigerator's built-in sensors and cameras will be used.

[0758] 6. Proposal message generation and transmission

[0759] The server identifies the necessary items based on the analysis results and the results of sentiment analysis, generates a suggested message for the user, and sends it to the device. The suggested message is customized according to the user's sentiment. A generative AI model (e.g., OpenAI GPT-3) is used to generate the suggested message.

[0760] 7. User Authorization and Automated Orders

[0761] The terminal receives the user's approval and sends the approval information to the server, which then automatically executes the order using the API of an e-commerce platform. Examples of such platforms include Amazon and Rakuten.

[0762] Specific operation example

[0763] For example, consider the case where one day a user tweets, "I'm running out of milk..." and there is a certain amount of impatience or anxiety in the voice.

[0764] 1. Audio collection

[0765] The user says, "I'm running out of milk..." The device collects this speech and sends it to the server.

[0766] 2. Voice Recognition

[0767] The server uses Google Cloud Speech-to-Text to convert the voice data into text data, which then becomes "We're running low on milk..."

[0768] 3. Natural Language Processing

[0769] The server uses SpaCy to analyze the text and extract the need "I need milk."

[0770] 4. Emotion analysis

[0771] The server uses IBM Watson Tone Analyzer to recognize the user's impatience or anxiety as emotions from the tone and tempo of the voice.

[0772] 5. Linking with sensor information

[0773] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[0774] 6. Proposal message generation and transmission

[0775] The server generates a suggested message saying, "You're running low on milk. Would you like to order some?" and, taking into account the user's impatience, suggests an additional message saying, "It looks like you need to order soon." These messages are generated using a generative AI model and sent to the device.

[0776] 7. User Authorization and Order Execution

[0777] The user sees the notification on their smartphone and responds verbally, "Yes, order." The device sends this approval to the server, which automatically fulfills the milk order using the e-commerce platform's API.

[0778] Prompt Sentence Examples

[0779] Here are some example prompts to input to the AI ​​model:

[0780] A user says, "Eggs are running low..." Use speech recognition technology to obtain the text data "Eggs are running low..." and use NLP to extract the need "I need eggs." Use an emotion engine to recognize the user's emotion (impatience or anxiety) from the tone of the voice, and combine this with egg inventory sensor information to generate a suggested message. The suggested message should include the basic message "We're running low on eggs. Would you like to order some?" as well as an additional message such as "It looks like you need to order soon." Finally, receive the user's approval and execute a flow that automatically orders eggs using the e-commerce platform's API.

[0781] As described above, by using the system of the present invention, necessary items can be automatically ordered quickly and efficiently based on the user's tweets, and by using the emotion engine, it is possible to make suggestions according to the user's feelings and situation. As a result, a system that can respond more appropriately and quickly to user needs can be realized.

[0782] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0783] Step 1: Audio Collection

[0784] One day, a user might say, "I'm running low on milk..." The microphone on the device collects the voice and transmits it to the server in real time. The noise-canceling function ensures clear voice data is transmitted. The input is the user's speech, and the output is digital voice data.

[0785] Step 2: Voice Recognition

[0786] The server converts the received voice data into text data using a speech recognition API such as Google Cloud Speech-to-Text. The input is voice data, and the output is the converted text data: "We're running out of milk..." This process converts the voice data into text format.

[0787] Step 3: Natural Language Processing (NLP)

[0788] The server passes the text data to a natural language processing framework (such as SpaCy or Google Cloud Natural Language API) for analysis. Through this analysis, the server extracts the user's need, "I need milk." The input is text data, and the output is the extracted need, "I need milk." The processing takes into account the context and identifies related keywords.

[0789] Step 4: Sentiment Analysis

[0790] The server passes the analyzed voice data to an emotion analysis engine (such as IBM Watson Tone Analyzer), which determines the user's emotion based on acoustic parameters such as tone, tempo, and volume. The input is the acoustic parameters of the voice data, and the output is a judgment result of the emotion, such as "impatience" or "anxiety."

[0791] Step 5: Linking with sensor information

[0792] The server acquires the inventory sensor information in the refrigerator and integrates it with the needs-based information. This information is collected from the inventory sensor in the refrigerator and retrieved using the smart refrigerator's API. The input is the needs and sensor information, and the output is the situation of "low milk stock."

[0793] Step 6: Generate and send a proposal message

[0794] The server generates a suggestion message based on the analysis results and sentiment analysis results. Using a generative AI model (e.g., OpenAI GPT-3), it generates the message "You're running low on milk. Would you like to order some?", plus an additional message such as "It looks like you need to order soon." The input is the need, sentiment analysis results, and sensor information, and the output is the generated suggestion message. The server sends the generated message to the device.

[0795] Step 7: User approval and order execution

[0796] The user confirms the notification on their smartphone and responds verbally with "Yes, please place the order." The device collects this approval and sends it to the server, which then automatically fulfills the milk order using the API of an e-commerce platform (e.g., Amazon or Rakuten Ichiba). The input is the user's approval, and the output is a confirmation message that the order has been fulfilled.

[0797] Through the above processing steps, the system can properly identify needs from the user's speech, generate and send a suggested message that takes into account the user's emotions and situation, and automatically order the item.

[0798] (Application example 2)

[0799] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0800] In modern society, users often find it difficult to order necessary items in a timely manner due to their busy daily lives. Furthermore, there is a lack of systems that provide appropriate suggestions based on the user's emotions and urgency. Conventional systems have difficulty accurately analyzing the user's emotions and making suggestions that reflect those emotions. Therefore, there is a need not only for systems that can automatically order necessary items based on the user's conversations and tweets, but also for systems that can customize the suggestions based on the user's emotions. Furthermore, there is a need to apply generative AI models to enhance the suggestions.

[0801] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0802] In this invention, the server includes: means for collecting user conversations using a voice input device; means for converting the collected voice data into text data; means for analyzing the text data using natural language processing (NLP) to determine the user's needs; means for reinforcing the analysis results by linking with other sensor information; means for identifying necessary items and generating and sending a suggested message to the user; means for placing an order on an e-commerce site or online sales platform based on the user's approval; means including an emotion analysis engine for analyzing the user's emotions from the conversation and reflecting the results in the suggested message; and means for inputting the suggested message as a prompt sentence into a generative AI model to reinforce the suggested content. This makes it possible to provide suggestions based on the user's needs and emotions, enabling the user to quickly and appropriately order the necessary items.

[0803] A "voice input device" is a device such as a microphone for collecting a user's conversations and tweets.

[0804] "Voice data" refers to data in which the user's voice is recorded in digital format.

[0805] "Character data" is data in text format obtained by analyzing voice data.

[0806] "Natural Language Processing (NLP)" is a technology for analyzing human language and understanding its meaning and intent.

[0807] "Sensor information" is, for example, environmental data obtained from inventory sensors and temperature / humidity sensors inside a refrigerator.

[0808] A "suggestion message" is a message containing a suggestion for ordering an item that is generated based on the user's needs and emotions.

[0809] An "emotion analysis engine" is a technology that analyzes a user's emotions from voice data and obtains the results.

[0810] A "generative AI model" is an artificial intelligence model that performs generative processing using large amounts of training data, and is used to enhance the content of suggestions.

[0811] A "prompt sentence" is an input sentence that gives instructions to a generative AI model.

[0812] An "ecommerce site or online sales platform" is a website or application that sells goods or services over the internet.

[0813] An "order fulfillment means" is a system or process for automatically ordering items based on user approval.

[0814] This invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. This system is realized using the following hardware and software.

[0815] System configuration

[0816] 1. Voice Input Device

[0817] The terminal is equipped with a built-in microphone and voice input device that constantly monitors the user's everyday conversations and tweets. This voice data is sent to a server via the Internet.

[0818] 2. Server Roles

[0819] The server is equipped with multiple software modules for performing the following processes:

[0820] Voice Recognition

[0821] We use voice recognition software (e.g., Google Cloud Speech-to-Text) to convert collected voice data into text data.

[0822] Natural Language Processing (NLP)

[0823] Use a natural language processing (NLP) framework (e.g., spaCy, NLTK) to analyze text data and determine user needs.

[0824] Sentiment Analysis Engine

[0825] An emotion analysis engine (e.g., IBM Watson) is used to analyze user emotions from voice data and obtain the results.

[0826] Linking with sensor information

[0827] It works in conjunction with other sensor information (e.g., inventory sensors and temperature / humidity sensors inside the refrigerator) to reinforce the analysis results.

[0828] Using generative AI models

[0829] The suggested message is input as a prompt into the generative AI model to reinforce the suggestion.

[0830] 3. Creating and sending a proposal message

[0831] The server generates a suggestion message based on the analysis results and sentiment analysis results and sends it to the user's device. The generated suggestion message is enhanced using the generative AI model.

[0832] 4. User Authorization and Automated Orders

[0833] The terminal receives the user's approval and sends the approval information to the server, which then automatically executes the order using the API of the e-commerce site or online sales platform.

[0834] Specific examples

[0835] For example, consider the case where the user says, "I'm a little tired today... I want to eat some delicious pizza." The processing flow in this case is as follows:

[0836] 1. User says: "I'm feeling a bit tired today... I want to eat some good pizza."

[0837] 2. Collecting voice data: The device's microphone collects voice and sends it to the server.

[0838] 3. Speech Recognition: Speech recognition software converts voice data into text data.

[0839] 4. Natural language processing: A natural language processing framework analyzes the text data and extracts the need, "I want to eat pizza."

[0840] 5. Sentiment analysis: The sentiment analysis engine recognizes the emotion "tired" from the user's speech.

[0841] 6. Generate suggestion message: The server generates a suggestion message such as, "You seem tired and want a nice pizza. Would you like us to order a pizza for delivery?" and sends it to the user's device.

[0842] 7. Use of generative AI model: The proposed message is input to the generative AI model as a prompt sentence and further enhanced. Specific examples of prompt sentences are as follows:

[0843] Prompt Sentence Examples

[0844] "A user said, 'I'm a little tired today... I want to eat some good pizza.' He seems tired. Please create a message suggesting delivery."

[0845] 8. User approval and order fulfillment: Once the user approves the proposed message, the server automatically fulfills the pizza order using the API of the e-commerce site or online sales platform.

[0846] As described above, the embodiment of the present invention has been specifically described. This system provides suggestions based on the user's needs and emotions, and enables the user to order goods quickly and appropriately.

[0847] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0848] Step 1:

[0849] (Audio collection)

[0850] The device uses a microphone to collect the user's speech. For example, the user might say, "I'm a little tired today... I want to eat some delicious pizza." The collected voice data is sent to a server via the Internet.

[0851] Input: User's voice

[0852] Output: Audio data

[0853] Step 2:

[0854] (Voice Recognition)

[0855] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the collected voice data into text data.

[0856] Input: Audio data

[0857] Output: Character data

[0858] Step 3:

[0859] (Natural Language Processing (NLP))

[0860] The server uses a natural language processing framework (e.g., spaCy, NLTK) to analyze the text data and determine the user's needs. Specifically, it extracts the need "I want pizza."

[0861] Input: Character data

[0862] Output: Needs data "I want to eat pizza"

[0863] Step 4:

[0864] (emotional analysis)

[0865] The server uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's emotions from the voice data. Specifically, it recognizes the emotion "tired" from the way the user speaks and the tone of their voice.

[0866] Input: Audio data

[0867] Output: Emotion data of "tired"

[0868] Step 5:

[0869] (Sensor information integration)

[0870] The server retrieves data from the refrigerator's inventory sensor to check whether pizza ingredients are available, and integrates the sensor information into the analysis results to reinforce the recommendation message.

[0871] Input: Refrigerator inventory information

[0872] Output: Consolidated data

[0873] Step 6:

[0874] (Generating a proposal message)

[0875] The server generates a suggestion message based on the need and emotion of "I want pizza, but I'm tired." Specifically, it creates a message that reads, "It seems you want to eat delicious pizza, but you're tired. Would you like us to order pizza for delivery?"

[0876] Input: Needs data such as "I want to eat pizza," emotion data such as "I'm tired," and sensor information

[0877] Output: Proposal message

[0878] Step 7:

[0879] (Using generative AI models)

[0880] The server provides a prompt to the generative AI model to enhance the suggestion message. An example of a prompt is, "The user said, 'I'm a little tired today... I want to eat some good pizza.' He seems tired. Please create a suggestion message that encourages delivery." An enhanced suggestion message is generated.

[0881] Input: Proposal message, prompt text

[0882] Output: Enhanced suggestion message

[0883] Step 8:

[0884] (Send a proposal message)

[0885] The server sends the enhanced proposal message to the terminal, and the user's terminal displays the proposal message for the user to confirm.

[0886] Input: Enhanced suggestion message

[0887] Output: The suggestion message that is displayed on the user's terminal.

[0888] Step 9:

[0889] (User approval and automatic ordering)

[0890] The user approves the proposed message (for example, by saying "Yes, please place the order"), and the device sends this approval to the server, which then executes the order using the API of the e-commerce site or online sales platform.

[0891] Input: User approval

[0892] Output: Order execution

[0893] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0894] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0895] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0896] [Third embodiment]

[0897] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0898] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0899] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0900] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0901] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0902] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0903] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0904] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0905] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0906] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0907] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0908] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0909] This invention is a system that utilizes user conversations and tweets to automatically order necessary items. This system collects and analyzes the user's voice, identifies the necessary items, makes ordering suggestions to the user, and, if the user approves, automatically places the order with an online supermarket or e-commerce site.

[0910] System configuration

[0911] 1. Voice Input Device

[0912] The terminal is equipped with a built-in voice input device equipped with a microphone, which constantly monitors the user's everyday conversations and tweets.

[0913] 2. Voice Recognition Technology

[0914] The server receives the voice data sent from the voice input device and converts the voice data into text data using voice recognition technology.

[0915] 3. Natural Language Processing (NLP)

[0916] The server analyzes the text data using natural language processing (NLP) to understand and identify the user's needs and requirements.

[0917] 4. Linking with sensor information

[0918] The server will then integrate other sensor information (e.g., inventory sensors and temperature and humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs.

[0919] 5. Creating and sending a proposal message

[0920] The server identifies the items the user needs based on the analysis results, generates a suggestion message for the user, and sends it to the user's device. The suggestion message is then delivered to the user via a smartphone notification or voice command.

[0921] 6. User Authorization and Automated Orders

[0922] The terminal receives the user's approval and sends that approval information to the server, which then uses the API of the online supermarket or e-commerce site to automatically place the order based on that information.

[0923] Specific operation example

[0924] For example, consider the case where a user tweets one day, "I'm running out of milk..."

[0925] 1. Audio collection

[0926] A user says, "I'm running out of milk..."

[0927] The device collects this audio and sends it to the server.

[0928] 2. Voice Recognition

[0929] The server converts the audio data into text data.

[0930] The text data obtained is "We're running out of milk..."

[0931] 3. Natural Language Processing

[0932] The server uses natural language processing to analyze the text and extract the need, "I need milk."

[0933] 4. Linking with sensor information

[0934] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[0935] 5. Creating and sending a proposal message

[0936] The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and sends it to the user's smartphone.

[0937] 6. User Authorization and Order Execution

[0938] The user checks the notification on their smartphone and responds verbally, "Yes, please place the order."

[0939] The device sends this approval to the server.

[0940] The server automatically executes the milk order using the online supermarket's API.

[0941] In this way, by using the system of the present invention, users can automatically order the items they need based on casual daily tweets, significantly reducing the effort required for shopping.

[0942] The processing flow will be explained below.

[0943] Step 1:

[0944] The voice input device constantly monitors the user's daily conversations and tweets. When voice data is input, the device collects the voice data and starts recording.

[0945] Step 2:

[0946] The device sends the collected voice data to the server in real time, and the voice data is securely transmitted over the network.

[0947] Step 3:

[0948] The server converts the received voice data into text data using a voice recognition engine, which uses a voice recognition algorithm.

[0949] Step 4:

[0950] The server passes the text data to a natural language processing (NLP) framework for analysis, which involves tokenizing the text into words, extracting nouns and verbs, and performing contextual analysis.

[0951] Step 5:

[0952] The server determines the user's intentions and needs from the analysis results. For example, if someone tweets "I'm running out of milk...", it will recognize that "I need milk."

[0953] Step 6:

[0954] The server checks the user's smart device and IoT sensor information, for example, checking the refrigerator's internal camera and inventory sensor to see if there is a low level of milk.

[0955] Step 7:

[0956] The server augments the analysis with sensor information and ultimately identifies the items needed. This specific step verifies that the sensor readings match expectations.

[0957] Step 8:

[0958] The server identifies the items needed and generates a suggestion message based on that, for example, "You're running low on milk. Would you like to order some?"

[0959] Step 9:

[0960] The server sends the generated proposal message to the user's terminal via a transmission protocol. The terminal notifies the user of the proposal message and waits for a response from the user.

[0961] Step 10:

[0962] The user reviews the suggestion message and takes an action to accept or reject it, for example, by saying "Yes, please place the order."

[0963] Step 11:

[0964] The device sends the user's authorization information to the server, using secure communication methods.

[0965] Step 12:

[0966] The server receives the authorization information and automatically orders the identified items using the API of the e-commerce site or online supermarket. Specifically, it generates an order request and sends it to the API endpoint.

[0967] Step 13:

[0968] The server retrieves the order confirmation and notifies the user that the order has been completed, generating detailed notifications along the way, including success or failure status.

[0969] This series of steps realizes a process for quickly and efficiently automatically ordering the necessary items from the user's spontaneous tweets.

[0970] Example 1

[0971] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0972] Conventional e-commerce systems require users to manually order the items they need each time, which is time-consuming and often leads to users forgetting to order. It is also difficult to grasp the user's situation and needs in real time and automate appropriate suggestions and orders based on that information. There is a need for a system that can solve these problems and enable users to purchase goods more conveniently and efficiently.

[0973] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0974] In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, and means for analyzing the text data using natural language processing (NLP) to determine the user's needs. This enables the server to automatically determine the items needed from the user's tweets and conversations, generate and send suggestion messages, and enable efficient purchasing.

[0975] "User" refers to an individual or business entity that uses the system.

[0976] "Audio input device" refers to a hardware device with a built-in microphone that collects a user's voice.

[0977] "Voice Data" refers to digitally recorded data of a user's voice collected by a voice input device.

[0978] "Character data" refers to text-format data converted from voice data using voice recognition technology.

[0979] "Natural Language Processing (NLP)" refers to the technology of analyzing text data to understand and identify user needs and requirements.

[0980] "Sensor information" refers to data obtained from IoT devices such as inventory sensors and temperature / humidity sensors inside the refrigerator.

[0981] A "suggestion message" refers to a message that is generated based on the analysis results and that suggests to the user to purchase an item.

[0982] "E-commerce system" refers to an online platform that enables the purchase of goods and services via the Internet.

[0983] "Dynamic pricing" refers to a pricing method in which prices fluctuate according to supply and demand.

[0984] "Approval information" refers to data provided by a user to indicate approval of a proposed message.

[0985] This invention is a system that automatically orders necessary items by utilizing user conversations and tweets. This system consists of a voice input device, i.e., a terminal equipped with a microphone for collecting the user's voice, and a server for processing this voice.

[0986] The device is equipped with a voice input device such as a smart speaker, which constantly collects the user's everyday conversations and tweets. For example, if a user tweets, "I'm running out of milk...", the voice data is encrypted and sent to a server in real time. The server then converts the voice data into text using voice recognition technologies such as the Google Speech-to-Text API or Azure Speech-to-Text.

[0987] The server then analyzes this text data using natural language processing (NLP) techniques such as BERT. This analysis determines the user's specific needs and requests, such as "I need milk." The server then checks the refrigerator's inventory, temperature, and humidity sensors to reinforce the analysis. Based on this information, the server determines whether the item the user needs is actually in short supply.

[0988] Once the server identifies the need, it generates a suggestion message saying, "You're low on milk. Would you like to order some?" This message is sent to the user via their smartphone or smart speaker. When the user responds to this suggestion with "Yes, please order," the device collects the voice and sends it back to the server.

[0989] Once the server confirms the user's approval, it uses the API of the e-commerce system to automatically place the order. For example, it automates the ordering process at online supermarkets and e-commerce sites, and payments are made using PayPal or credit card information. Once the order is complete, a confirmation email is sent to the user.

[0990] Specific examples

[0991] Example 1: When you are running out of milk

[0992] 1. A user tweets, "I'm running out of milk..."

[0993] 2. The device collects the audio and sends it to the server.

[0994] 3. The server uses the Google Speech-to-Text API to convert the audio data into text data.

[0995] 4. The server uses BERT to extract the need "I need milk."

[0996] 5. The server retrieves data from the refrigerator's inventory sensor and checks for low inventory.

[0997] 6. The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and sends it to the smartphone.

[0998] 7. The user responds, "Yes, place the order."

[0999] 8. The device sends the authorization information to the server.

[1000] 9. The server automatically orders milk using the API of the e-commerce system.

[1001] This system allows users to automatically order the items they need based on casual daily tweets, significantly reducing the effort required for shopping.

[1002] Examples of prompt statements

[1003] "Please tell me how to collect tweets about how I'm running out of milk, determine their needs using natural language processing, and send them suggestion messages."

[1004] "Please tell me the specific steps of a system that uses voice recognition technology to identify the items needed from the user's voice and automatically places an order with an online supermarket."

[1005] As described above, the present invention utilizes users' conversations and tweets, and makes use of voice input devices and advanced analysis technology to make users' lives more convenient.

[1006] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1007] Step 1: Collect audio input

[1008] The device uses a built-in microphone to constantly collect the user's everyday conversations and tweets. The input is the user's voice, which is converted into digital voice data. Specifically, when a user tweets, "I'm running out of milk...", the device collects the voice in real time. The output is the collected voice data.

[1009] Step 2: Sending audio data

[1010] The voice data collected by the terminal is encrypted and sent to the server via the Internet. The input is the digital voice data collected in step 1, and the output is the voice data sent to the server. Specifically, the terminal converts the collected voice data into packets and sends them to the server.

[1011] Step 3: Voice Recognition

[1012] The server uses speech recognition technology such as Google Speech-to-Text API or Azure Speech-to-Text to convert the voice data into text data. The input is the voice data sent in step 2, and the output is text data. Specifically, the server analyzes the voice data and generates text data such as "I'm running low on milk..."

[1013] Step 4: Natural Language Processing (NLP)

[1014] The server uses natural language processing (NLP) techniques such as BERT to analyze the text data and identify the user's needs. The input is the text data generated in step 3, and the output is the user's identified need (e.g., "I need milk"). Specifically, the server analyzes the text data and extracts the need "I need milk" from "I'm running low on milk..."

[1015] Step 5: Acquire sensor information

[1016] The server obtains data from the inventory sensor and other sensors in the refrigerator to augment the analysis results. The input is data from the inventory sensor and temperature / humidity sensors in the refrigerator, and the output is the confirmed inventory status (e.g., "Milk is low in stock"). Specifically, the server obtains information that "Milk is low in stock" from the inventory sensor in the refrigerator.

[1017] Step 6: Proposal message generation

[1018] The server generates a suggestion message for the user based on the analysis results and sensor information. The input is the user's needs and sensor information, and the output is the generated suggestion message (e.g., "You're running low on milk. Would you like to order some?"). Specifically, the server creates a suggestion message based on the information that "milk is needed" and "stock is low."

[1019] Step 7: Sending a proposal message

[1020] The server generates a proposal message and sends it to the user's smartphone or smart speaker. The input is the proposal message generated in step 6, and the output is the message sent to the user's device. The specific operation is to send the message generated by the server to the user's smartphone as a notification.

[1021] Step 8: Receive user approval

[1022] The device collects the user's approval information and sends it to the server. The input is the user's approval (e.g., "Yes, please order"), and the output is the approval information sent to the server. Specifically, the user checks the notification on their smartphone and responds by saying "Yes, please order." The device collects this approval information and sends it to the server.

[1023] Step 9: Automatic Order Execution

[1024] The server uses the API of the e-commerce system to automatically place an order based on the user's approval. The input is the user's approval information and suggested product information, and the output is the completed order information. Specifically, the server uses the API of the online supermarket to automatically place an order for milk and send a confirmation email to the user.

[1025] Through each of the above processing steps, this system can automatically order the necessary items based on the user's conversation, significantly reducing the effort required for shopping.

[1026] (Application example 1)

[1027] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1028] In everyday life, users often have to go through the trouble of ordering food or other items. Particularly in food delivery, users have to go through the trouble of opening an app to place an order and searching for specific food options. Therefore, to improve user convenience, there is a demand for a system that automatically orders the necessary items or meals by utilizing the user's everyday conversations and tweets.

[1029] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1030] In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, means for analyzing the text data using natural language processing (NLP) to determine the user's needs, means for reinforcing the analysis results in conjunction with other sensor information, means for identifying necessary items and generating and sending suggested messages to the user, means for placing orders with e-commerce sites or online supermarkets based on the user's approval, and means for automatically placing orders using a food delivery API. This enables users to easily order items and meals through their everyday tweets and conversations.

[1031] "User" refers to any individual or corporation that uses this system.

[1032] A "conversation" refers to any or all of a user's spoken communication.

[1033] An "audio input device" is a device for collecting a user's voice, and includes, for example, a smartphone or smart glasses.

[1034] "Audio Data" refers to analog or digital audio signals collected by an audio input device.

[1035] "Text data" refers to text data converted from voice data using voice recognition technology.

[1036] "Natural language processing (NLP)" refers to technology that analyzes text data to understand its meaning and intent and identify user needs.

[1037] "Needs" refer to goods and services that users desire.

[1038] "Sensor Information" refers to information obtained from sensors that provide data related to a user or their environment.

[1039] "Suggestion Message" refers to a notification or message generated by the system to suggest desired goods or services to a user.

[1040] "Acceptance" refers to a positive response made by a user to a suggestion message.

[1041] "EC site" refers to a website for conducting electronic commerce.

[1042] An "online supermarket" refers to a website that sells groceries and daily necessities online.

[1043] A "food delivery API" refers to an interface that uses specific protocols and formats to exchange data between food delivery services and systems.

[1044] "Program" refers to a set of instructions that are executed on a computer system.

[1045] System configuration

[1046] A system for implementing the present invention includes the following components:

[1047] 1. Voice Input Device

[1048] A voice input device is a device equipped with a microphone for collecting a user's voice. Specifically, it includes a smartphone and smart glasses. This allows the device to collect a user's everyday conversations and tweets.

[1049] 2. Voice Recognition Technology

[1050] Voice data collected by the voice input device is sent to a cloud-based server, which is equipped with voice recognition technology (such as Google Cloud Speech-to-Text) and converts the voice data into text data.

[1051] 3. Natural Language Processing (NLP)

[1052] The server analyzes the converted text data using natural language processing (NLP) techniques (e.g., Transformer-based NLP models, BERT, GPT, etc.), and the analysis identifies the user's needs.

[1053] 4. Linking with sensor information

[1054] The server also integrates information provided by the refrigerator and indoor environmental sensors to augment the analysis results and more accurately identify the user's needs.

[1055] 5. Creating and sending a proposal message

[1056] The server generates a suggestion message based on the identified user needs, which is then sent as a notification to the user's smartphone, smart glasses, or other device, such as a message like, "Would you like to order curry?"

[1057] 6. User Authorization and Automated Orders

[1058] The user can approve the proposed message by voice or tap. This approval information is sent back to the server, and the order is automatically placed via the food delivery service's API (e.g., Uber Eats API).

[1059] Specific operation example

[1060] 1. Users tweet using voice

[1061] The user whispers into their smart glasses or smartphone, "I want to eat curry today."

[1062] 2. The audio data is sent to the server

[1063] The voice input device collects the user's voice and transmits the voice data to a server in real time.

[1064] 3. Generating text data using speech recognition

[1065] The server receives the voice data and converts it into text data such as "I'd like to eat curry today" using voice recognition technology such as Google Cloud Speech-to-Text.

[1066] 4. Identifying needs through natural language processing

[1067] The server analyzes this text data using natural language processing technology and identifies the need, such as "I need curry."

[1068] 5. Creating and Sending a Proposal Message

[1069] The server generates a suggestion message saying "Would you like to order curry?" and sends it to the user's smart device.

[1070] 6. User Authorization and Order Execution

[1071] When the user responds, "Yes, please order," the approval information is sent to the server, and the curry order is automatically placed through the food delivery API.

[1072] Example prompt

[1073] When a user says "I want to eat curry," generate a prompt that extracts the user's need to order "curry" as a result of natural language processing.

[1074] In this way, by using the system of the present invention, users can smoothly order meals from casual daily tweets, which is expected to significantly improve the convenience of users' lives.

[1075] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1076] Step 1:

[1077] The user speaks a specified phrase into the smart glasses or smartphone, which then collects the user's speech through the voice input device and generates audio data. The input is the user's voice, and the output is an audio file of the collected raw data.

[1078] Step 2:

[1079] The device collects audio data and sends it to the server in real time. The input is an audio file, and the output is the audio data sent to the server. The device uses the microphone and internet connection for audio input.

[1080] Step 3:

[1081] The server converts the received voice data into text using voice recognition technology such as Google Cloud Speech-to-Text. The input is voice data, and the converted text data is the output. This step converts the voice signal into text.

[1082] Step 4:

[1083] The server uses natural language processing (NLP) technology to analyze text data and identify user needs. The input is text data, and the analysis results, or needs information, are output. Here, a generative AI model (e.g., BERT or GPT) is used.

[1084] Step 5:

[1085] The server connects with sensor information obtained from the refrigerator and indoor environment sensors to augment the analysis results. The input is the needs information identified by NLP and sensor information, and the output is the augmented needs information. IoT devices are used to collect the sensor information.

[1086] Step 6:

[1087] The server generates a suggestion message based on the augmented needs information and sends it to the user's smart device. The input is the augmented needs information, and the output is the generated suggestion message. Here, an advanced text generation algorithm is used to generate the message.

[1088] Step 7:

[1089] The user receives the suggestion message, confirms it, and approves it by voice or tap. The input is the suggestion message, and the output is the user's approval information. The user's device plays a key role in this step.

[1090] Step 8:

[1091] The terminal sends the user's authorization information to the server. The input is the user's authorization information, and the output is the data sent to the server. The authorization information is again sent to the server via the Internet.

[1092] Step 9:

[1093] The server uses the food delivery service's API to automatically execute the order. The input is the user's approval information and needs information, and the output is a notification to the food delivery service that the order has been completed. The order is processed through communication using the API.

[1094] Examples of prompts used

[1095] When a user says "I want to eat curry," generate a prompt that extracts the user's need to order "curry" as a result of natural language processing.

[1096] In this way, the process from the user's voice input to the completion of the order is carried out smoothly, allowing the user to order the necessary goods or meals with less hassle.

[1097] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1098] The present invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. Specific embodiments of the system are described in detail below.

[1099] System configuration

[1100] 1. Voice Input Device

[1101] The terminal is equipped with a built-in voice input device equipped with a microphone, which constantly monitors the user's everyday conversations and tweets.

[1102] 2. Voice Recognition Technology

[1103] The server receives the voice data sent from the voice input device and converts the voice data into text data using voice recognition technology.

[1104] 3. Natural Language Processing (NLP)

[1105] The server passes the text data to a natural language processing (NLP) framework for analysis, which understands and identifies the user's needs and requirements.

[1106] 4. Linking with sensor information

[1107] The server will then integrate other sensor information (e.g., inventory sensors and temperature and humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs.

[1108] 5. Emotion Engine

[1109] The server is equipped with an emotion engine that analyzes emotions from the user's voice data, determining the user's emotions not only from the results of natural language processing but also from the tone, tempo, and other acoustic parameters of the voice.

[1110] 6. Proposal message generation and transmission

[1111] The server identifies the necessary items based on the analysis results and the emotion analysis results, and generates and sends a suggestion message to the user. The suggestion message is customized according to the user's emotion.

[1112] 7. User Authorization and Automated Orders

[1113] The terminal receives the user's approval and sends the approval information to the server, which then automatically places the order using the API of the online supermarket or e-commerce site based on the information.

[1114] Specific operation example

[1115] For example, consider the case where one day a user tweets, "I'm running out of milk..." and there is a certain amount of impatience or anxiety in the voice.

[1116] 1. Audio collection

[1117] A user says, "I'm running out of milk..."

[1118] The device collects this audio and sends it to the server.

[1119] 2. Voice Recognition

[1120] The server converts the audio data into text data.

[1121] The text data obtained is "We're running out of milk..."

[1122] 3. Natural Language Processing

[1123] The server uses natural language processing to analyze the text and extract the need, "I need milk."

[1124] 4. Emotion analysis

[1125] The server uses an emotion engine to recognize the user's impatience or anxiety as emotions from the voice data.

[1126] 5. Linking with sensor information

[1127] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[1128] 6. Proposal message generation and transmission

[1129] The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and, taking into account the user's impatience, also suggests an additional message saying, "It looks like you'll need to order soon."

[1130] The server sends a proposal message to the user's terminal.

[1131] 7. User Authorization and Order Execution

[1132] The user checks the notification on their smartphone and responds verbally, "Yes, please place the order."

[1133] The device sends this approval to the server.

[1134] The server automatically executes the milk order using the online supermarket's API.

[1135] In this way, by using the system of the present invention, necessary items can be ordered automatically and quickly and efficiently based on the user's tweets, and by using the emotion engine, it is possible to make suggestions according to the user's feelings and situation. As a result, a system that can respond more appropriately and quickly to user needs is realized.

[1136] The processing flow will be explained below.

[1137] Step 1:

[1138] The voice input device constantly monitors the user's daily conversations and tweets. When voice data is input, the device collects the voice data and starts recording.

[1139] Step 2:

[1140] The device sends the collected voice data to the server in real time, and the voice data is securely transmitted over the network.

[1141] Step 3:

[1142] The server converts the received voice data into text data using a speech recognition engine. A speech recognition algorithm is used in this process. For example, a voice saying "I'm running out of milk..." is converted into text data.

[1143] Step 4:

[1144] The server passes the text data to a natural language processing (NLP) framework for analysis. The analysis involves tokenizing the text, extracting nouns and verbs, and performing contextual analysis. This allows the need, "I need milk," to be extracted.

[1145] Step 5:

[1146] The server uses an emotion engine to analyze the user's emotions from the voice data. In this process, the server determines the user's emotions (e.g., impatience, joy, anxiety, etc.) from the tone, tempo, and other acoustic parameters of the voice. For example, if impatience is detected, the user is judged to be in a hurry.

[1147] Step 6:

[1148] The server determines the user's needs based on the analysis results and obtains sensor information from the refrigerator to confirm whether the identified items are truly necessary. For example, it checks whether there is a low amount of milk remaining using a camera or inventory sensor inside the refrigerator.

[1149] Step 7:

[1150] The server augments the analysis results with sensor information and ultimately identifies the item needed (in this case, milk).

[1151] Step 8:

[1152] The server generates a suggested message for the user, for example, "You're running low on milk. Would you like to order some?", and includes an additional message, "It looks like you need to order soon," taking into account the results of sentiment analysis (impatience).

[1153] Step 9:

[1154] The server sends the generated proposal message to the user's terminal, which notifies the user of the proposal message and waits for the user's response.

[1155] Step 10:

[1156] The user confirms the suggestion message and responds by voice, "Yes, I would like to order." Here, the user responds using a smartphone or voice input device.

[1157] Step 11:

[1158] The device sends the user's authorization information to the server, using secure communication methods.

[1159] Step 12:

[1160] The server receives the authorization information and automatically orders the specified item (milk) using the API of the online supermarket or e-commerce site. Specifically, it generates an order request and sends it to the API endpoint.

[1161] Step 13:

[1162] The server retrieves the order confirmation and notifies the user that the order has been completed, generating detailed notifications along the way, including success or failure status.

[1163] This series of steps not only enables a process for quickly and efficiently automatically ordering the necessary items based on the user's natural tweets and conversations, but also makes it possible to use an emotion engine to make optimal suggestions according to the situation.

[1164] Example 2

[1165] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1166] Conventional systems using voice input devices have difficulty accurately recognizing users' conversations and tweets and automatically ordering items. Furthermore, they are unable to make suggestions that take the user's emotions into account, making it difficult to place an order at the appropriate time. Furthermore, because they do not consider the user's level of urgency or emotions, the suggestions they make may not meet the user's needs.

[1167] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, means for analyzing the text data using natural language processing to determine the user's needs, means for reinforcing the analysis results in cooperation with other sensor information, means for determining the user's emotions using an emotion analysis engine, means for identifying required items and generating and transmitting a suggestion message taking the user's emotions into consideration, and means for placing an order with the e-commerce platform based on the user's approval. This makes it possible to appropriately identify needs from the user's conversations and tweets, generate a suggestion message taking the user's emotions and urgency into consideration, and automatically order items.

[1168] A "voice input device" is a hardware device that collects a user's speech and transmits it to a server as digital voice data.

[1169] "Voice data" is data that represents the user's speech in digital form.

[1170] "Character data" is data in text format converted from voice data.

[1171] "Natural language processing" is a technology that enables computers to understand and analyze human language.

[1172] "Sensor information" is environmental or condition data obtained from another device, such as temperature, humidity, or inventory status.

[1173] An "emotion analysis engine" is software that analyzes the tone and tempo of a user's speech and infers the user's emotions from that speech.

[1174] A "suggestion message" is a message generated by the system to suggest an item order to the user.

[1175] An "e-commerce platform" is an internet service that allows you to order and purchase products online.

[1176] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to automatically generate text or suggested messages.

[1177] "Server" means the central computing unit of the system, a computer used to analyze voice data, perform sentiment analysis and collaboration with other data, and execute orders for the necessary goods.

[1178] "User conversation" refers to the utterances and tweets that users make in their daily lives.

[1179] "Approval" is the act of the user agreeing to a proposed message from the system.

[1180] The present invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. Specific embodiments of the system are described in detail below.

[1181] System configuration

[1182] 1. Voice Input Device

[1183] The device is equipped with a built-in microphone and voice input device that constantly monitors the user's daily conversations and tweets. For example, a microphone with a noise-canceling function is used.

[1184] 2. Voice Recognition Technology

[1185] The server receives the voice data sent from the voice input device and converts it into text data using voice recognition technology, possibly using APIs such as Google Cloud Speech-to-Text.

[1186] 3. Natural Language Processing (NLP)

[1187] The server passes the text data to a natural language processing (NLP) framework for analysis, which then uses SpaCy and the Google Cloud Natural Language API to understand and identify the user's needs and requirements.

[1188] 4. Sentiment Analysis Engine

[1189] The server is equipped with an emotion analysis engine that analyzes emotions from the user's voice data. The emotion analysis engine determines the user's emotions not only from the results of natural language processing, but also from the tone, tempo, and other acoustic parameters of the voice. It uses a system such as IBM Watson Tone Analyzer.

[1190] 5. Linking with sensor information

[1191] The server will then link with other sensor information (e.g., inventory sensors and temperature / humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs. The smart refrigerator's built-in sensors and cameras will be used.

[1192] 6. Proposal message generation and transmission

[1193] The server identifies the necessary items based on the analysis results and the results of sentiment analysis, generates a suggested message for the user, and sends it to the device. The suggested message is customized according to the user's sentiment. A generative AI model (e.g., OpenAI GPT-3) is used to generate the suggested message.

[1194] 7. User Authorization and Automated Orders

[1195] The terminal receives the user's approval and sends the approval information to the server, which then automatically executes the order using the API of an e-commerce platform. Examples of such platforms include Amazon and Rakuten Ichiba.

[1196] Specific operation example

[1197] For example, consider the case where one day a user tweets, "I'm running out of milk..." and there is a certain amount of impatience or anxiety in the voice.

[1198] 1. Audio collection

[1199] The user says, "I'm running out of milk..." The device collects this speech and sends it to the server.

[1200] 2. Voice Recognition

[1201] The server uses Google Cloud Speech-to-Text to convert the voice data into text data, which then becomes "We're running low on milk..."

[1202] 3. Natural Language Processing

[1203] The server uses SpaCy to analyze the text and extract the need "I need milk."

[1204] 4. Emotion analysis

[1205] The server uses IBM Watson Tone Analyzer to recognize the user's impatience or anxiety as emotions from the tone and tempo of the voice.

[1206] 5. Linking with sensor information

[1207] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[1208] 6. Proposal message generation and transmission

[1209] The server generates a suggested message saying, "You're running low on milk. Would you like to order some?" and, taking into account the user's impatience, suggests an additional message saying, "It looks like you need to order soon." These messages are generated using a generative AI model and sent to the device.

[1210] 7. User Authorization and Order Execution

[1211] The user sees the notification on their smartphone and responds verbally, "Yes, order." The device sends this approval to the server, which automatically fulfills the milk order using the e-commerce platform's API.

[1212] Prompt Sentence Examples

[1213] Here are some example prompts to input to the AI ​​model:

[1214] A user says, "Eggs are running low..." Use speech recognition technology to obtain the text data "Eggs are running low..." and use NLP to extract the need "I need eggs." Use an emotion engine to recognize the user's emotion (impatience or anxiety) from the tone of the voice, and combine this with egg inventory sensor information to generate a suggested message. The suggested message should include the basic message "We're running low on eggs. Would you like to order some?" as well as an additional message such as "It looks like you need to order soon." Finally, receive the user's approval and execute a flow that automatically orders eggs using the e-commerce platform's API.

[1215] As described above, by using the system of the present invention, necessary items can be automatically ordered quickly and efficiently based on the user's tweets, and by using the emotion engine, it is possible to make suggestions according to the user's feelings and situation. As a result, a system that can respond more appropriately and quickly to user needs can be realized.

[1216] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1217] Step 1: Audio Collection

[1218] One day, a user might say, "I'm running low on milk..." The microphone on the device collects the voice and transmits it to the server in real time. The noise-canceling function ensures clear voice data is transmitted. The input is the user's speech, and the output is digital voice data.

[1219] Step 2: Voice Recognition

[1220] The server converts the received voice data into text data using a speech recognition API such as Google Cloud Speech-to-Text. The input is voice data, and the output is the converted text data: "We're running out of milk..." This process converts the voice data into text format.

[1221] Step 3: Natural Language Processing (NLP)

[1222] The server passes the text data to a natural language processing framework (such as SpaCy or Google Cloud Natural Language API) for analysis. Through this analysis, the server extracts the user's need, "I need milk." The input is text data, and the output is the extracted need, "I need milk." The processing takes into account the context and identifies related keywords.

[1223] Step 4: Sentiment Analysis

[1224] The server passes the analyzed voice data to an emotion analysis engine (such as IBM Watson Tone Analyzer), which determines the user's emotion based on acoustic parameters such as tone, tempo, and volume. The input is the acoustic parameters of the voice data, and the output is a judgment result of the emotion, such as "impatience" or "anxiety."

[1225] Step 5: Linking with sensor information

[1226] The server acquires the inventory sensor information in the refrigerator and integrates it with the needs-based information. This information is collected from the inventory sensor in the refrigerator and retrieved using the smart refrigerator's API. The input is the needs and sensor information, and the output is the situation of "low milk stock."

[1227] Step 6: Generate and send a proposal message

[1228] The server generates a suggestion message based on the analysis results and sentiment analysis results. Using a generative AI model (e.g., OpenAI GPT-3), it generates the message "You're running low on milk. Would you like to order some?", plus an additional message such as "It looks like you need to order soon." The input is the need, sentiment analysis results, and sensor information, and the output is the generated suggestion message. The server sends the generated message to the device.

[1229] Step 7: User approval and order execution

[1230] The user confirms the notification on their smartphone and responds verbally with "Yes, please place the order." The device collects this approval and sends it to the server, which then automatically fulfills the milk order using the API of an e-commerce platform (e.g., Amazon or Rakuten Ichiba). The input is the user's approval, and the output is a confirmation message that the order has been fulfilled.

[1231] Through the above processing steps, the system can properly identify needs from the user's speech, generate and send a suggested message that takes into account the user's emotions and situation, and automatically order the item.

[1232] (Application example 2)

[1233] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1234] In modern society, users often find it difficult to order necessary items in a timely manner due to their busy daily lives. Furthermore, there is a lack of systems that provide appropriate suggestions based on the user's emotions and urgency. Conventional systems have difficulty accurately analyzing the user's emotions and making suggestions that reflect those emotions. Therefore, there is a need not only for systems that can automatically order necessary items based on the user's conversations and tweets, but also for systems that can customize the suggestions based on the user's emotions. Furthermore, there is a need to apply generative AI models to enhance the suggestions.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1236] In this invention, the server includes: means for collecting user conversations using a voice input device; means for converting the collected voice data into text data; means for analyzing the text data using natural language processing (NLP) to determine the user's needs; means for reinforcing the analysis results by linking with other sensor information; means for identifying necessary items and generating and sending a suggested message to the user; means for placing an order on an e-commerce site or online sales platform based on the user's approval; means including an emotion analysis engine for analyzing the user's emotions from the conversation and reflecting the results in the suggested message; and means for inputting the suggested message as a prompt sentence into a generative AI model to reinforce the suggested content. This makes it possible to provide suggestions based on the user's needs and emotions, enabling the user to quickly and appropriately order the necessary items.

[1237] A "voice input device" is a device such as a microphone for collecting a user's conversations and tweets.

[1238] "Voice data" refers to data in which the user's voice is recorded in digital format.

[1239] "Character data" is data in text format obtained by analyzing voice data.

[1240] "Natural Language Processing (NLP)" is a technology for analyzing human language and understanding its meaning and intent.

[1241] "Sensor information" is, for example, environmental data obtained from inventory sensors and temperature / humidity sensors inside a refrigerator.

[1242] A "suggestion message" is a message containing a suggestion for ordering an item that is generated based on the user's needs and emotions.

[1243] An "emotion analysis engine" is a technology that analyzes a user's emotions from voice data and obtains the results.

[1244] A "generative AI model" is an artificial intelligence model that performs generative processing using large amounts of training data, and is used to enhance the content of suggestions.

[1245] A "prompt sentence" is an input sentence that gives instructions to a generative AI model.

[1246] An "ecommerce site or online sales platform" is a website or application that sells goods or services over the internet.

[1247] An "order fulfillment means" is a system or process for automatically ordering items based on user approval.

[1248] This invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. This system is realized using the following hardware and software.

[1249] System configuration

[1250] 1. Voice Input Device

[1251] The terminal is equipped with a built-in microphone and voice input device that constantly monitors the user's everyday conversations and tweets. This voice data is sent to a server via the Internet.

[1252] 2. Server Roles

[1253] The server is equipped with multiple software modules for performing the following processes:

[1254] Voice Recognition

[1255] We use voice recognition software (e.g., Google Cloud Speech-to-Text) to convert collected voice data into text data.

[1256] Natural Language Processing (NLP)

[1257] Use a natural language processing (NLP) framework (e.g., spaCy, NLTK) to analyze text data and determine user needs.

[1258] Sentiment Analysis Engine

[1259] An emotion analysis engine (e.g., IBM Watson) is used to analyze user emotions from voice data and obtain the results.

[1260] Linking with sensor information

[1261] It works in conjunction with other sensor information (e.g., inventory sensors and temperature / humidity sensors inside the refrigerator) to reinforce the analysis results.

[1262] Using generative AI models

[1263] The suggested message is input as a prompt into the generative AI model to reinforce the suggestion.

[1264] 3. Creating and sending a proposal message

[1265] The server generates a suggestion message based on the analysis results and sentiment analysis results and sends it to the user's device. The generated suggestion message is enhanced using the generative AI model.

[1266] 4. User Authorization and Automated Orders

[1267] The terminal receives the user's approval and sends the approval information to the server, which then automatically executes the order using the API of the e-commerce site or online sales platform.

[1268] Specific examples

[1269] For example, consider the case where the user says, "I'm a little tired today... I want to eat some delicious pizza." The processing flow in this case is as follows:

[1270] 1. User says: "I'm feeling a bit tired today... I want to eat some good pizza."

[1271] 2. Collecting voice data: The device's microphone collects voice and sends it to the server.

[1272] 3. Speech Recognition: Speech recognition software converts voice data into text data.

[1273] 4. Natural language processing: A natural language processing framework analyzes the text data and extracts the need, "I want to eat pizza."

[1274] 5. Sentiment analysis: The sentiment analysis engine recognizes the emotion "tired" from the user's speech.

[1275] 6. Generate suggestion message: The server generates a suggestion message such as, "You seem tired and want a nice pizza. Would you like us to order a pizza for delivery?" and sends it to the user's device.

[1276] 7. Use of generative AI model: The proposed message is input to the generative AI model as a prompt sentence and further enhanced. Specific examples of prompt sentences are as follows:

[1277] Prompt Sentence Examples

[1278] "A user said, 'I'm a little tired today... I want to eat some good pizza.' He seems tired. Please create a message suggesting delivery."

[1279] 8. User approval and order fulfillment: Once the user approves the proposed message, the server automatically fulfills the pizza order using the API of the e-commerce site or online sales platform.

[1280] As described above, the embodiment of the present invention has been specifically described. This system provides suggestions based on the user's needs and emotions, and enables the user to order goods quickly and appropriately.

[1281] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1282] Step 1:

[1283] (Audio collection)

[1284] The device uses a microphone to collect the user's speech. For example, the user might say, "I'm a little tired today... I want to eat some delicious pizza." The collected voice data is sent to a server via the Internet.

[1285] Input: User's voice

[1286] Output: Audio data

[1287] Step 2:

[1288] (Voice Recognition)

[1289] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the collected voice data into text data.

[1290] Input: Audio data

[1291] Output: Character data

[1292] Step 3:

[1293] (Natural Language Processing (NLP))

[1294] The server uses a natural language processing framework (e.g., spaCy, NLTK) to analyze the text data and determine the user's needs. Specifically, it extracts the need "I want pizza."

[1295] Input: Character data

[1296] Output: Needs data "I want to eat pizza"

[1297] Step 4:

[1298] (emotional analysis)

[1299] The server uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's emotions from the voice data. Specifically, it recognizes the emotion "tired" from the way the user speaks and the tone of their voice.

[1300] Input: Audio data

[1301] Output: Emotion data of "tired"

[1302] Step 5:

[1303] (Sensor information integration)

[1304] The server retrieves data from the refrigerator's inventory sensor to check whether pizza ingredients are available, and integrates the sensor information into the analysis results to reinforce the recommendation message.

[1305] Input: Refrigerator inventory information

[1306] Output: Consolidated data

[1307] Step 6:

[1308] (Generating a proposal message)

[1309] The server generates a suggestion message based on the need and emotion of "I want pizza, but I'm tired." Specifically, it creates a message that reads, "It seems you want to eat delicious pizza, but you're tired. Would you like us to order pizza for delivery?"

[1310] Input: Needs data such as "I want to eat pizza," emotion data such as "I'm tired," and sensor information

[1311] Output: Proposal message

[1312] Step 7:

[1313] (Using generative AI models)

[1314] The server provides a prompt to the generative AI model to enhance the suggestion message. An example of a prompt is, "The user said, 'I'm a little tired today... I want to eat some good pizza.' He seems tired. Please create a suggestion message that encourages delivery." An enhanced suggestion message is generated.

[1315] Input: Proposal message, prompt text

[1316] Output: Enhanced suggestion message

[1317] Step 8:

[1318] (Send a proposal message)

[1319] The server sends the enhanced proposal message to the terminal, and the user's terminal displays the proposal message for the user to confirm.

[1320] Input: Enhanced suggestion message

[1321] Output: The suggestion message that is displayed on the user's terminal.

[1322] Step 9:

[1323] (User approval and automatic ordering)

[1324] The user approves the proposed message (for example, by saying "Yes, please place the order"), and the device sends this approval to the server, which then executes the order using the API of the e-commerce site or online sales platform.

[1325] Input: User approval

[1326] Output: Order execution

[1327] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1328] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1329] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1330] [Fourth embodiment]

[1331] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1332] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1333] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1334] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1335] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1336] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1337] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1338] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1339] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1340] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1341] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1342] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1343] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1344] This invention is a system that utilizes user conversations and tweets to automatically order necessary items. This system collects and analyzes the user's voice, identifies the necessary items, makes ordering suggestions to the user, and, if the user approves, automatically places the order with an online supermarket or e-commerce site.

[1345] System configuration

[1346] 1. Voice Input Device

[1347] The terminal is equipped with a built-in voice input device equipped with a microphone, which constantly monitors the user's everyday conversations and tweets.

[1348] 2. Voice Recognition Technology

[1349] The server receives the voice data sent from the voice input device and converts the voice data into text data using voice recognition technology.

[1350] 3. Natural Language Processing (NLP)

[1351] The server analyzes the text data using natural language processing (NLP) to understand and identify the user's needs and requirements.

[1352] 4. Linking with sensor information

[1353] The server will then integrate other sensor information (e.g., inventory sensors and temperature and humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs.

[1354] 5. Creating and sending a proposal message

[1355] The server identifies the items the user needs based on the analysis results, generates a suggestion message for the user, and sends it to the user's device. The suggestion message is then delivered to the user via a smartphone notification or voice command.

[1356] 6. User Authorization and Automated Orders

[1357] The terminal receives the user's approval and sends that approval information to the server, which then uses the API of the online supermarket or e-commerce site to automatically place the order based on that information.

[1358] Specific operation example

[1359] For example, consider the case where a user tweets one day, "I'm running out of milk..."

[1360] 1. Audio collection

[1361] A user says, "I'm running out of milk..."

[1362] The device collects this audio and sends it to the server.

[1363] 2. Voice Recognition

[1364] The server converts the audio data into text data.

[1365] The text data obtained is "We're running out of milk..."

[1366] 3. Natural Language Processing

[1367] The server uses natural language processing to analyze the text and extract the need, "I need milk."

[1368] 4. Linking with sensor information

[1369] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[1370] 5. Creating and sending a proposal message

[1371] The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and sends it to the user's smartphone.

[1372] 6. User Authorization and Order Execution

[1373] The user checks the notification on their smartphone and responds verbally, "Yes, please place the order."

[1374] The device sends this approval to the server.

[1375] The server automatically executes the milk order using the online supermarket's API.

[1376] In this way, by using the system of the present invention, users can automatically order the items they need based on casual daily tweets, significantly reducing the effort required for shopping.

[1377] The processing flow will be explained below.

[1378] Step 1:

[1379] The voice input device constantly monitors the user's daily conversations and tweets. When voice data is input, the device collects the voice data and starts recording.

[1380] Step 2:

[1381] The device sends the collected voice data to the server in real time, and the voice data is securely transmitted over the network.

[1382] Step 3:

[1383] The server converts the received voice data into text data using a voice recognition engine, which uses a voice recognition algorithm.

[1384] Step 4:

[1385] The server passes the text data to a natural language processing (NLP) framework for analysis, which involves tokenizing the text into words, extracting nouns and verbs, and performing contextual analysis.

[1386] Step 5:

[1387] The server determines the user's intentions and needs from the analysis results. For example, if someone tweets "I'm running out of milk...", it will recognize that "I need milk."

[1388] Step 6:

[1389] The server checks the user's smart device and IoT sensor information, for example, checking the refrigerator's internal camera and inventory sensor to see if there is a low level of milk.

[1390] Step 7:

[1391] The server augments the analysis with sensor information and ultimately identifies the items needed. This specific step verifies that the sensor readings match expectations.

[1392] Step 8:

[1393] The server identifies the items needed and generates a suggestion message based on that, for example, "You're running low on milk. Would you like to order some?"

[1394] Step 9:

[1395] The server sends the generated proposal message to the user's terminal via a transmission protocol. The terminal notifies the user of the proposal message and waits for a response from the user.

[1396] Step 10:

[1397] The user reviews the suggestion message and takes an action to accept or reject it, for example, by saying "Yes, please place the order."

[1398] Step 11:

[1399] The device sends the user's authorization information to the server, using secure communication methods.

[1400] Step 12:

[1401] The server receives the authorization information and automatically orders the identified items using the API of the e-commerce site or online supermarket. Specifically, it generates an order request and sends it to the API endpoint.

[1402] Step 13:

[1403] The server retrieves the order confirmation and notifies the user that the order has been completed, generating detailed notifications along the way, including success or failure status.

[1404] This series of steps realizes a process for quickly and efficiently automatically ordering the necessary items from the user's spontaneous tweets.

[1405] Example 1

[1406] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1407] Conventional e-commerce systems require users to manually order the items they need each time, which is time-consuming and often leads to users forgetting to order. It is also difficult to grasp the user's situation and needs in real time and automate appropriate suggestions and orders based on that information. There is a need for a system that can solve these problems and enable users to purchase goods more conveniently and efficiently.

[1408] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1409] In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, and means for analyzing the text data using natural language processing (NLP) to determine the user's needs. This enables the server to automatically determine the items needed from the user's tweets and conversations, generate and send suggestion messages, and enable efficient purchasing.

[1410] "User" refers to an individual or business entity that uses the system.

[1411] "Audio input device" refers to a hardware device with a built-in microphone that collects a user's voice.

[1412] "Voice Data" refers to digitally recorded data of a user's voice collected by a voice input device.

[1413] "Character data" refers to text-format data converted from voice data using voice recognition technology.

[1414] "Natural Language Processing (NLP)" refers to the technology of analyzing text data to understand and identify user needs and requirements.

[1415] "Sensor information" refers to data obtained from IoT devices such as inventory sensors and temperature / humidity sensors inside the refrigerator.

[1416] A "suggestion message" refers to a message that is generated based on the analysis results and that suggests to the user to purchase an item.

[1417] "E-commerce system" refers to an online platform that enables the purchase of goods and services via the Internet.

[1418] "Dynamic pricing" refers to a pricing method in which prices fluctuate according to supply and demand.

[1419] "Approval information" refers to data provided by a user to indicate approval of a proposed message.

[1420] This invention is a system that automatically orders necessary items by utilizing user conversations and tweets. This system consists of a voice input device, i.e., a terminal with a built-in microphone for collecting the user's voice, and a server for processing this.

[1421] The device is equipped with a voice input device such as a smart speaker, which constantly collects the user's everyday conversations and tweets. For example, if a user tweets, "I'm running out of milk...", the voice data is encrypted and sent to a server in real time. The server then converts the voice data into text using voice recognition technologies such as the Google Speech-to-Text API or Azure Speech-to-Text.

[1422] The server then analyzes this text data using natural language processing (NLP) techniques such as BERT. This analysis determines the user's specific needs and requests, such as "I need milk." The server then checks the refrigerator's inventory, temperature, and humidity sensors to reinforce the analysis. Based on this information, the server determines whether the item the user needs is actually in short supply.

[1423] Once the server identifies the need, it generates a suggestion message saying, "You're low on milk. Would you like to order some?" This message is sent to the user via their smartphone or smart speaker. When the user responds to this suggestion with "Yes, please order," the device collects the voice and sends it back to the server.

[1424] Once the server confirms the user's approval, it uses the API of the e-commerce system to automatically place the order. For example, it automates the ordering process at online supermarkets and e-commerce sites, and payments are made using PayPal or credit card information. Once the order is complete, a confirmation email is sent to the user.

[1425] Specific examples

[1426] Example 1: When you are running out of milk

[1427] 1. A user tweets, "I'm running out of milk..."

[1428] 2. The device collects the audio and sends it to the server.

[1429] 3. The server uses the Google Speech-to-Text API to convert the audio data into text data.

[1430] 4. The server uses BERT to extract the need "I need milk."

[1431] 5. The server retrieves data from the refrigerator's inventory sensor and checks for low inventory.

[1432] 6. The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and sends it to the smartphone.

[1433] 7. The user responds, "Yes, place the order."

[1434] 8. The device sends the authorization information to the server.

[1435] 9. The server automatically orders milk using the API of the e-commerce system.

[1436] This system allows users to automatically order the items they need based on casual daily tweets, significantly reducing the effort required for shopping.

[1437] Examples of prompt statements

[1438] "Please tell me how to collect tweets about how I'm running out of milk, determine their needs using natural language processing, and send them suggestion messages."

[1439] "Please tell me the specific steps of a system that uses voice recognition technology to identify the items needed from the user's voice and automatically places an order with an online supermarket."

[1440] As described above, the present invention utilizes users' conversations and tweets, and makes use of voice input devices and advanced analysis technology to make users' lives more convenient.

[1441] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1442] Step 1: Collect audio input

[1443] The device uses a built-in microphone to constantly collect the user's everyday conversations and tweets. The input is the user's voice, which is converted into digital voice data. Specifically, when a user tweets, "I'm running out of milk...", the device collects the voice in real time. The output is the collected voice data.

[1444] Step 2: Sending audio data

[1445] The voice data collected by the terminal is encrypted and sent to the server via the Internet. The input is the digital voice data collected in step 1, and the output is the voice data sent to the server. Specifically, the terminal converts the collected voice data into packets and sends them to the server.

[1446] Step 3: Voice Recognition

[1447] The server uses speech recognition technology such as Google Speech-to-Text API or Azure Speech-to-Text to convert the voice data into text data. The input is the voice data sent in step 2, and the output is text data. Specifically, the server analyzes the voice data and generates text data such as "I'm running low on milk..."

[1448] Step 4: Natural Language Processing (NLP)

[1449] The server uses natural language processing (NLP) techniques such as BERT to analyze the text data and identify the user's needs. The input is the text data generated in step 3, and the output is the user's identified need (e.g., "I need milk"). Specifically, the server analyzes the text data and extracts the need "I need milk" from "I'm running low on milk..."

[1450] Step 5: Acquire sensor information

[1451] The server obtains data from the inventory sensor and other sensors in the refrigerator to augment the analysis results. The input is data from the inventory sensor and temperature / humidity sensors in the refrigerator, and the output is the confirmed inventory status (e.g., "Milk is low in stock"). Specifically, the server obtains information that "Milk is low in stock" from the inventory sensor in the refrigerator.

[1452] Step 6: Proposal message generation

[1453] The server generates a suggestion message for the user based on the analysis results and sensor information. The input is the user's needs and sensor information, and the output is the generated suggestion message (e.g., "You're running low on milk. Would you like to order some?"). Specifically, the server creates a suggestion message based on the information that "milk is needed" and "stock is low."

[1454] Step 7: Sending a proposal message

[1455] The server generates a proposal message and sends it to the user's smartphone or smart speaker. The input is the proposal message generated in step 6, and the output is the message sent to the user's device. The specific operation is to send the message generated by the server to the user's smartphone as a notification.

[1456] Step 8: Receive user approval

[1457] The device collects the user's approval information and sends it to the server. The input is the user's approval (e.g., "Yes, please order"), and the output is the approval information sent to the server. Specifically, the user checks the notification on their smartphone and responds by saying "Yes, please order." The device collects this approval information and sends it to the server.

[1458] Step 9: Automatic Order Execution

[1459] The server uses the API of the e-commerce system to automatically place an order based on the user's approval. The input is the user's approval information and suggested product information, and the output is the completed order information. Specifically, the server uses the API of the online supermarket to automatically place an order for milk and send a confirmation email to the user.

[1460] Through each of the above processing steps, this system can automatically order the necessary items based on the user's conversation, significantly reducing the effort required for shopping.

[1461] (Application example 1)

[1462] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1463] In everyday life, users often have to go through the trouble of ordering food or other items. Particularly in food delivery, users have to go through the trouble of opening an app to place an order and searching for specific food options. Therefore, to improve user convenience, there is a demand for a system that automatically orders the necessary items or meals by utilizing the user's everyday conversations and tweets.

[1464] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1465] In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, means for analyzing the text data using natural language processing (NLP) to determine the user's needs, means for reinforcing the analysis results in conjunction with other sensor information, means for identifying necessary items and generating and sending suggested messages to the user, means for placing orders with e-commerce sites or online supermarkets based on the user's approval, and means for automatically placing orders using a food delivery API. This enables users to easily order items and meals through their everyday tweets and conversations.

[1466] "User" refers to any individual or corporation that uses this system.

[1467] A "conversation" refers to any or all of a user's spoken communication.

[1468] An "audio input device" is a device for collecting a user's voice, and includes, for example, a smartphone or smart glasses.

[1469] "Audio Data" refers to analog or digital audio signals collected by an audio input device.

[1470] "Text data" refers to text data converted from voice data using voice recognition technology.

[1471] "Natural language processing (NLP)" refers to technology that analyzes text data to understand its meaning and intent and identify user needs.

[1472] "Needs" refer to goods and services that users desire.

[1473] "Sensor Information" refers to information obtained from sensors that provide data related to a user or their environment.

[1474] "Suggestion Message" refers to a notification or message generated by the system to suggest desired goods or services to a user.

[1475] "Acceptance" refers to a positive response made by a user to a suggestion message.

[1476] "EC site" refers to a website for conducting electronic commerce.

[1477] An "online supermarket" refers to a website that sells groceries and daily necessities online.

[1478] A "food delivery API" refers to an interface that uses specific protocols and formats to exchange data between food delivery services and systems.

[1479] "Program" refers to a set of instructions that are executed on a computer system.

[1480] System configuration

[1481] A system for implementing the present invention includes the following components:

[1482] 1. Voice Input Device

[1483] A voice input device is a device equipped with a microphone for collecting a user's voice. Specifically, it includes a smartphone and smart glasses. This allows the device to collect a user's everyday conversations and tweets.

[1484] 2. Voice Recognition Technology

[1485] The voice data collected by the voice input device is sent to a cloud-based server, which is equipped with voice recognition technology (such as Google Cloud Speech-to-Text) and converts the voice data into text data.

[1486] 3. Natural Language Processing (NLP)

[1487] The server analyzes the converted text data using natural language processing (NLP) techniques (e.g., Transformer-based NLP models, BERT, GPT, etc.), and the analysis identifies the user's needs.

[1488] 4. Linking with sensor information

[1489] The server also integrates information provided by the refrigerator and indoor environmental sensors to augment the analysis results and more accurately identify the user's needs.

[1490] 5. Creating and sending a proposal message

[1491] The server generates a suggestion message based on the identified user needs, which is then sent as a notification to the user's smartphone, smart glasses, or other device, such as a message like, "Would you like to order curry?"

[1492] 6. User Authorization and Automated Orders

[1493] The user can approve the proposed message by voice or tap. This approval information is sent back to the server, and the order is automatically placed via the food delivery service's API (e.g., Uber Eats API).

[1494] Specific operation example

[1495] 1. Users tweet using voice

[1496] The user whispers into their smart glasses or smartphone, "I want to eat curry today."

[1497] 2. The audio data is sent to the server

[1498] The voice input device collects the user's voice and transmits the voice data to a server in real time.

[1499] 3. Generating text data using speech recognition

[1500] The server receives the voice data and converts it into text data such as "I'd like to eat curry today" using voice recognition technology such as Google Cloud Speech-to-Text.

[1501] 4. Identifying needs through natural language processing

[1502] The server analyzes this text data using natural language processing technology and identifies the need, such as "I need curry."

[1503] 5. Creating and Sending a Proposal Message

[1504] The server generates a suggestion message saying "Would you like to order curry?" and sends it to the user's smart device.

[1505] 6. User Authorization and Order Execution

[1506] When the user responds, "Yes, please order," the approval information is sent to the server, and the curry order is automatically placed through the food delivery API.

[1507] Example prompt

[1508] When a user says "I want to eat curry," generate a prompt that extracts the user's need to order "curry" as a result of natural language processing.

[1509] In this way, by using the system of the present invention, users can smoothly order meals from casual daily tweets, which is expected to significantly improve the convenience of users' lives.

[1510] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1511] Step 1:

[1512] The user speaks a specified phrase into the smart glasses or smartphone, which then collects the user's speech through the voice input device and generates audio data. The input is the user's voice, and the output is an audio file of the collected raw data.

[1513] Step 2:

[1514] The device collects audio data and sends it to the server in real time. The input is an audio file, and the output is the audio data sent to the server. The device uses the microphone and internet connection for audio input.

[1515] Step 3:

[1516] The server converts the received voice data into text using voice recognition technology such as Google Cloud Speech-to-Text. The input is voice data, and the converted text data is the output. This step converts the voice signal into text.

[1517] Step 4:

[1518] The server uses natural language processing (NLP) technology to analyze text data and identify user needs. The input is text data, and the analysis results, or needs information, are output. Here, a generative AI model (e.g., BERT or GPT) is used.

[1519] Step 5:

[1520] The server connects with sensor information obtained from the refrigerator and indoor environment sensors to augment the analysis results. The input is the needs information identified by NLP and sensor information, and the output is the augmented needs information. IoT devices are used to collect the sensor information.

[1521] Step 6:

[1522] The server generates a suggestion message based on the augmented needs information and sends it to the user's smart device. The input is the augmented needs information, and the output is the generated suggestion message. Here, an advanced text generation algorithm is used to generate the message.

[1523] Step 7:

[1524] The user receives the suggestion message, confirms it, and approves it by voice or tap. The input is the suggestion message, and the output is the user's approval information. The user's device plays a key role in this step.

[1525] Step 8:

[1526] The terminal sends the user's authorization information to the server. The input is the user's authorization information, and the output is the data sent to the server. The authorization information is again sent to the server via the Internet.

[1527] Step 9:

[1528] The server uses the food delivery service's API to automatically execute the order. The input is the user's approval information and needs information, and the output is a notification to the food delivery service that the order has been completed. The order is processed through communication using the API.

[1529] Examples of prompts used

[1530] When a user says "I want to eat curry," generate a prompt that extracts the user's need to order "curry" as a result of natural language processing.

[1531] In this way, the process from the user's voice input to the completion of the order is carried out smoothly, allowing the user to order the necessary goods or meals with less hassle.

[1532] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1533] The present invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. Specific embodiments of the system are described in detail below.

[1534] System configuration

[1535] 1. Voice Input Device

[1536] The terminal is equipped with a built-in voice input device equipped with a microphone, which constantly monitors the user's everyday conversations and tweets.

[1537] 2. Voice Recognition Technology

[1538] The server receives the voice data sent from the voice input device and converts the voice data into text data using voice recognition technology.

[1539] 3. Natural Language Processing (NLP)

[1540] The server passes the text data to a natural language processing (NLP) framework for analysis, which understands and identifies the user's needs and requirements.

[1541] 4. Linking with sensor information

[1542] The server will then integrate other sensor information (e.g., inventory sensors and temperature and humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs.

[1543] 5. Emotion Engine

[1544] The server is equipped with an emotion engine that analyzes emotions from the user's voice data, determining the user's emotions not only from the results of natural language processing but also from the tone, tempo, and other acoustic parameters of the voice.

[1545] 6. Proposal message generation and transmission

[1546] The server identifies the necessary items based on the analysis results and the emotion analysis results, and generates and sends a suggestion message to the user. The suggestion message is customized according to the user's emotion.

[1547] 7. User Authorization and Automated Orders

[1548] The terminal receives the user's approval and sends the approval information to the server, which then automatically places the order using the API of the online supermarket or e-commerce site based on the information.

[1549] Specific operation example

[1550] For example, consider the case where one day a user tweets, "I'm running out of milk..." and there is a certain amount of impatience or anxiety in the voice.

[1551] 1. Audio collection

[1552] A user says, "I'm running out of milk..."

[1553] The device collects this audio and sends it to the server.

[1554] 2. Voice Recognition

[1555] The server converts the audio data into text data.

[1556] The text data obtained is "We're running out of milk..."

[1557] 3. Natural Language Processing

[1558] The server uses natural language processing to analyze the text and extract the need, "I need milk."

[1559] 4. Emotion analysis

[1560] The server uses an emotion engine to recognize the user's impatience or anxiety as emotions from the voice data.

[1561] 5. Linking with sensor information

[1562] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[1563] 6. Proposal message generation and transmission

[1564] The server generates a suggestion message saying, "You're running low on milk. Would you like to order some?" and, taking into account the user's impatience, also suggests an additional message saying, "It looks like you'll need to order soon."

[1565] The server sends a proposal message to the user's terminal.

[1566] 7. User Authorization and Order Execution

[1567] The user checks the notification on their smartphone and responds verbally, "Yes, please place the order."

[1568] The device sends this approval to the server.

[1569] The server automatically executes the milk order using the online supermarket's API.

[1570] In this way, by using the system of the present invention, necessary items can be ordered automatically and quickly and efficiently based on the user's tweets, and by using the emotion engine, it is possible to make suggestions according to the user's feelings and situation. As a result, a system that can respond more appropriately and quickly to user needs is realized.

[1571] The processing flow will be explained below.

[1572] Step 1:

[1573] The voice input device constantly monitors the user's daily conversations and tweets. When voice data is input, the device collects the voice data and starts recording.

[1574] Step 2:

[1575] The device sends the collected voice data to the server in real time, and the voice data is securely transmitted over the network.

[1576] Step 3:

[1577] The server converts the received voice data into text data using a speech recognition engine. A speech recognition algorithm is used in this process. For example, a voice saying "I'm running out of milk..." is converted into text data.

[1578] Step 4:

[1579] The server passes the text data to a natural language processing (NLP) framework for analysis. The analysis involves tokenizing the text, extracting nouns and verbs, and performing contextual analysis. This allows the need, "I need milk," to be extracted.

[1580] Step 5:

[1581] The server uses an emotion engine to analyze the user's emotions from the voice data. In this process, the server determines the user's emotions (e.g., impatience, joy, anxiety, etc.) from the tone, tempo, and other acoustic parameters of the voice. For example, if impatience is detected, the user is judged to be in a hurry.

[1582] Step 6:

[1583] The server determines the user's needs based on the analysis results and obtains sensor information from the refrigerator to confirm whether the identified items are truly necessary. For example, it checks whether there is a low amount of milk remaining using a camera or inventory sensor inside the refrigerator.

[1584] Step 7:

[1585] The server augments the analysis results with sensor information and ultimately identifies the item needed (in this case, milk).

[1586] Step 8:

[1587] The server generates a suggested message for the user, for example, "You're running low on milk. Would you like to order some?", and includes an additional message, "It looks like you need to order soon," taking into account the results of sentiment analysis (impatience).

[1588] Step 9:

[1589] The server sends the generated proposal message to the user's terminal, which notifies the user of the proposal message and waits for the user's response.

[1590] Step 10:

[1591] The user confirms the suggestion message and responds by voice, "Yes, I would like to order." Here, the user responds using a smartphone or voice input device.

[1592] Step 11:

[1593] The device sends the user's authorization information to the server, using secure communication methods.

[1594] Step 12:

[1595] The server receives the authorization information and automatically orders the specified item (milk) using the API of the online supermarket or e-commerce site. Specifically, it generates an order request and sends it to the API endpoint.

[1596] Step 13:

[1597] The server retrieves the order confirmation and notifies the user that the order has been completed, generating detailed notifications along the way, including success or failure status.

[1598] This series of steps not only enables a process for quickly and efficiently automatically ordering the necessary items based on the user's natural tweets and conversations, but also makes it possible to use an emotion engine to make optimal suggestions according to the situation.

[1599] Example 2

[1600] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1601] Conventional systems using voice input devices have difficulty accurately recognizing users' conversations and tweets and automatically ordering items. Furthermore, they are unable to make suggestions that take the user's emotions into account, making it difficult to place an order at the appropriate time. Furthermore, because they do not consider the user's level of urgency or emotions, the suggestions they make may not meet the user's needs.

[1602] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user conversations using a voice input device, means for converting the collected voice data into text data, means for analyzing the text data using natural language processing to determine the user's needs, means for reinforcing the analysis results in cooperation with other sensor information, means for determining the user's emotions using an emotion analysis engine, means for identifying required items and generating and transmitting a suggestion message taking the user's emotions into consideration, and means for placing an order with the e-commerce platform based on the user's approval. This makes it possible to appropriately identify needs from the user's conversations and tweets, generate a suggestion message taking the user's emotions and urgency into consideration, and automatically order items.

[1603] A "voice input device" is a hardware device that collects a user's speech and transmits it to a server as digital voice data.

[1604] "Voice data" is data that represents the user's speech in digital form.

[1605] "Character data" is data in text format converted from voice data.

[1606] "Natural language processing" is a technology that enables computers to understand and analyze human language.

[1607] "Sensor information" is environmental or condition data obtained from another device, such as temperature, humidity, or inventory status.

[1608] An "emotion analysis engine" is software that analyzes the tone and tempo of a user's speech and infers the user's emotions from that speech.

[1609] A "suggestion message" is a message generated by the system to suggest an item order to the user.

[1610] An "e-commerce platform" is an internet service that allows you to order and purchase products online.

[1611] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to automatically generate text or suggested messages.

[1612] "Server" means the central computing unit of the system, a computer used to analyze voice data, perform sentiment analysis and collaboration with other data, and execute orders for the necessary goods.

[1613] "User conversation" refers to the utterances and tweets that users make in their daily lives.

[1614] "Approval" is the act of the user agreeing to a proposed message from the system.

[1615] The present invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. Specific embodiments of the system are described in detail below.

[1616] System configuration

[1617] 1. Voice Input Device

[1618] The device is equipped with a built-in microphone and voice input device that constantly monitors the user's daily conversations and tweets. For example, a microphone with a noise-canceling function is used.

[1619] 2. Voice Recognition Technology

[1620] The server receives the voice data sent from the voice input device and converts it into text data using voice recognition technology, possibly using APIs such as Google Cloud Speech-to-Text.

[1621] 3. Natural Language Processing (NLP)

[1622] The server passes the text data to a natural language processing (NLP) framework for analysis, which then uses SpaCy and the Google Cloud Natural Language API to understand and identify the user's needs and requirements.

[1623] 4. Sentiment Analysis Engine

[1624] The server is equipped with an emotion analysis engine that analyzes emotions from the user's voice data. The emotion analysis engine determines the user's emotions not only from the results of natural language processing, but also from the tone, tempo, and other acoustic parameters of the voice. It uses a system such as IBM Watson Tone Analyzer.

[1625] 5. Linking with sensor information

[1626] The server will then link with other sensor information (e.g., inventory sensors and temperature / humidity sensors inside the refrigerator) to augment the analysis results. This information will be used to confirm or augment the identified needs. The smart refrigerator's built-in sensors and cameras will be used.

[1627] 6. Proposal message generation and transmission

[1628] The server identifies the necessary items based on the analysis results and the results of sentiment analysis, generates a suggested message for the user, and sends it to the device. The suggested message is customized according to the user's sentiment. A generative AI model (e.g., OpenAI GPT-3) is used to generate the suggested message.

[1629] 7. User Authorization and Automated Orders

[1630] The terminal receives the user's approval and sends the approval information to the server, which then automatically executes the order using the API of an e-commerce platform. Examples of such platforms include Amazon and Rakuten.

[1631] Specific operation example

[1632] For example, consider the case where one day a user tweets, "I'm running out of milk..." and there is a certain amount of impatience or anxiety in the voice.

[1633] 1. Audio collection

[1634] The user says, "I'm running out of milk..." The device collects this speech and sends it to the server.

[1635] 2. Voice Recognition

[1636] The server uses Google Cloud Speech-to-Text to convert the voice data into text data, which then becomes "We're running low on milk..."

[1637] 3. Natural Language Processing

[1638] The server uses SpaCy to analyze the text and extract the need "I need milk."

[1639] 4. Emotion analysis

[1640] The server uses IBM Watson Tone Analyzer to recognize the user's impatience or anxiety as emotions from the tone and tempo of the voice.

[1641] 5. Linking with sensor information

[1642] The server checks the refrigerator sensor information and realizes that milk is low in stock.

[1643] 6. Proposal message generation and transmission

[1644] The server generates a suggested message saying, "You're running low on milk. Would you like to order some?" and, taking into account the user's impatience, suggests an additional message saying, "It looks like you need to order soon." These messages are generated using a generative AI model and sent to the device.

[1645] 7. User Authorization and Order Execution

[1646] The user sees the notification on their smartphone and responds verbally, "Yes, order." The device sends this approval to the server, which automatically fulfills the milk order using the e-commerce platform's API.

[1647] Prompt Sentence Examples

[1648] Here are some example prompts to input to the AI ​​model:

[1649] A user says, "Eggs are running low..." Use speech recognition technology to obtain the text data "Eggs are running low..." and use NLP to extract the need "I need eggs." Use an emotion engine to recognize the user's emotion (impatience or anxiety) from the tone of the voice, and combine this with egg inventory sensor information to generate a suggested message. The suggested message should include the basic message "We're running low on eggs. Would you like to order some?" as well as an additional message such as "It looks like you need to order soon." Finally, receive the user's approval and execute a flow that automatically orders eggs using the e-commerce platform's API.

[1650] As described above, by using the system of the present invention, necessary items can be automatically ordered quickly and efficiently based on the user's tweets, and by using the emotion engine, it is possible to make suggestions according to the user's feelings and situation. As a result, a system that can respond more appropriately and quickly to user needs can be realized.

[1651] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1652] Step 1: Audio Collection

[1653] One day, a user might say, "I'm running low on milk..." The microphone on the device collects the voice and transmits it to the server in real time. The noise-canceling function ensures clear voice data is transmitted. The input is the user's speech, and the output is digital voice data.

[1654] Step 2: Voice Recognition

[1655] The server converts the received voice data into text data using a speech recognition API such as Google Cloud Speech-to-Text. The input is voice data, and the output is the converted text data: "We're running out of milk..." This process converts the voice data into text format.

[1656] Step 3: Natural Language Processing (NLP)

[1657] The server passes the text data to a natural language processing framework (such as SpaCy or Google Cloud Natural Language API) for analysis. Through this analysis, the server extracts the user's need, "I need milk." The input is text data, and the output is the extracted need, "I need milk." The processing takes into account the context and identifies related keywords.

[1658] Step 4: Sentiment Analysis

[1659] The server passes the analyzed voice data to an emotion analysis engine (such as IBM Watson Tone Analyzer), which determines the user's emotion based on acoustic parameters such as tone, tempo, and volume. The input is the acoustic parameters of the voice data, and the output is a judgment result of the emotion, such as "impatience" or "anxiety."

[1660] Step 5: Linking with sensor information

[1661] The server acquires the inventory sensor information in the refrigerator and integrates it with the needs-based information. This information is collected from the inventory sensor in the refrigerator and retrieved using the smart refrigerator's API. The input is the needs and sensor information, and the output is the situation of "low milk stock."

[1662] Step 6: Generate and send a proposal message

[1663] The server generates a suggestion message based on the analysis results and sentiment analysis results. Using a generative AI model (e.g., OpenAI GPT-3), it generates the message "You're running low on milk. Would you like to order some?", plus an additional message such as "It looks like you need to order soon." The input is the need, sentiment analysis results, and sensor information, and the output is the generated suggestion message. The server sends the generated message to the device.

[1664] Step 7: User approval and order execution

[1665] The user confirms the notification on their smartphone and responds verbally with "Yes, please place the order." The device collects this approval and sends it to the server, which then automatically fulfills the milk order using the API of an e-commerce platform (e.g., Amazon or Rakuten Ichiba). The input is the user's approval, and the output is a confirmation message that the order has been fulfilled.

[1666] Through the above processing steps, the system can properly identify needs from the user's speech, generate and send a suggested message that takes into account the user's emotions and situation, and automatically order the item.

[1667] (Application example 2)

[1668] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1669] In modern society, users often find it difficult to order necessary items in a timely manner due to their busy daily lives. Furthermore, there is a lack of systems that provide appropriate suggestions based on the user's emotions and urgency. Conventional systems have difficulty accurately analyzing the user's emotions and making suggestions that reflect those emotions. Therefore, there is a need not only for systems that can automatically order necessary items based on the user's conversations and tweets, but also for systems that can customize the suggestions based on the user's emotions. Furthermore, there is a need to apply generative AI models to enhance the suggestions.

[1670] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1671] In this invention, the server includes: means for collecting user conversations using a voice input device; means for converting the collected voice data into text data; means for analyzing the text data using natural language processing (NLP) to determine the user's needs; means for reinforcing the analysis results by linking with other sensor information; means for identifying necessary items and generating and sending a suggested message to the user; means for placing an order on an e-commerce site or online sales platform based on the user's approval; means including an emotion analysis engine for analyzing the user's emotions from the conversation and reflecting the results in the suggested message; and means for inputting the suggested message as a prompt sentence into a generative AI model to reinforce the suggested content. This makes it possible to provide suggestions based on the user's needs and emotions, enabling the user to quickly and appropriately order the necessary items.

[1672] A "voice input device" is a device such as a microphone for collecting a user's conversations and tweets.

[1673] "Voice data" refers to data in which the user's voice is recorded in digital format.

[1674] "Character data" is data in text format obtained by analyzing voice data.

[1675] "Natural Language Processing (NLP)" is a technology for analyzing human language and understanding its meaning and intent.

[1676] "Sensor information" is, for example, environmental data obtained from inventory sensors and temperature / humidity sensors inside a refrigerator.

[1677] A "suggestion message" is a message containing a suggestion for ordering an item that is generated based on the user's needs and emotions.

[1678] An "emotion analysis engine" is a technology that analyzes a user's emotions from voice data and obtains the results.

[1679] A "generative AI model" is an artificial intelligence model that performs generative processing using large amounts of training data, and is used to enhance the content of suggestions.

[1680] A "prompt sentence" is an input sentence that gives instructions to a generative AI model.

[1681] An "ecommerce site or online sales platform" is a website or application that sells goods or services over the internet.

[1682] An "order fulfillment means" is a system or process for automatically ordering items based on user approval.

[1683] This invention is a system that combines a system that automatically orders necessary items using user conversations and tweets with an emotion engine that recognizes the user's emotions and reflects them in the recommendations. This system is realized using the following hardware and software.

[1684] System configuration

[1685] 1. Voice Input Device

[1686] The terminal is equipped with a built-in microphone and voice input device that constantly monitors the user's everyday conversations and tweets. This voice data is sent to a server via the Internet.

[1687] 2. Server Roles

[1688] The server is equipped with multiple software modules for performing the following processes:

[1689] Voice Recognition

[1690] We use voice recognition software (e.g., Google Cloud Speech-to-Text) to convert collected voice data into text data.

[1691] Natural Language Processing (NLP)

[1692] Use a natural language processing (NLP) framework (e.g., spaCy, NLTK) to analyze text data and determine user needs.

[1693] Sentiment Analysis Engine

[1694] An emotion analysis engine (e.g., IBM Watson) is used to analyze user emotions from voice data and obtain the results.

[1695] Linking with sensor information

[1696] It works in conjunction with other sensor information (e.g., inventory sensors and temperature / humidity sensors inside the refrigerator) to reinforce the analysis results.

[1697] Using generative AI models

[1698] The suggested message is input as a prompt into the generative AI model to reinforce the suggestion.

[1699] 3. Creating and sending a proposal message

[1700] The server generates a suggestion message based on the analysis results and sentiment analysis results and sends it to the user's device. The generated suggestion message is enhanced using the generative AI model.

[1701] 4. User Authorization and Automated Orders

[1702] The terminal receives the user's approval and sends the approval information to the server, which then automatically executes the order using the API of the e-commerce site or online sales platform.

[1703] Specific examples

[1704] For example, consider the case where the user says, "I'm a little tired today... I want to eat some delicious pizza." The processing flow in this case is as follows:

[1705] 1. User says: "I'm feeling a bit tired today... I want to eat some good pizza."

[1706] 2. Collecting voice data: The device's microphone collects voice and sends it to the server.

[1707] 3. Speech Recognition: Speech recognition software converts voice data into text data.

[1708] 4. Natural language processing: A natural language processing framework analyzes the text data and extracts the need, "I want to eat pizza."

[1709] 5. Sentiment analysis: The sentiment analysis engine recognizes the emotion "tired" from the user's speech.

[1710] 6. Generate suggestion message: The server generates a suggestion message such as, "You seem tired and want a nice pizza. Would you like us to order a pizza for delivery?" and sends it to the user's device.

[1711] 7. Use of generative AI model: The proposed message is input to the generative AI model as a prompt sentence and further enhanced. Specific examples of prompt sentences are as follows:

[1712] Prompt Sentence Examples

[1713] The user said, "I'm a bit tired today... I really want to eat delicious pizza." He seems to be tired. Please create a proposal message to encourage delivery.

[1714] 8. User Approval and Order Execution: When the user approves the proposal message, the server automatically executes the pizza order using the APIs of the e-commerce site or online sales platform.

[1715] As described above, the embodiments for implementing this invention have been specifically described. With this system, it is possible to provide proposals based on user needs and emotions and to order items quickly and appropriately.

[1716] The flow of specific processing in Application Example 2 will be described using FIG. 14.

[1717] Step 1:

[1718] (Voice Collection)

[1719] The terminal collects the user's conversation using the microphone. The user says, "I'm a bit tired today... I really want to eat delicious pizza." The collected voice data is transmitted to the server through the Internet.

[1720] Input: User's voice

[1721] Output: Voice data

[1722] Step 2:

[1723] (Voice Recognition)

[1724] The server uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert the collected voice data into character data.

[1725] Input: Voice data

[1726] Output: Character data

[1727] Step 3:

[1728] (Natural Language Processing (NLP))

[1729] The server uses a natural language processing framework (e.g., spaCy, NLTK) to analyze the text data and determine the user's needs. Specifically, it extracts the need "I want pizza."

[1730] Input: Character data

[1731] Output: Needs data "I want to eat pizza"

[1732] Step 4:

[1733] (emotional analysis)

[1734] The server uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's emotions from the voice data. Specifically, it recognizes the emotion "tired" from the way the user speaks and the tone of their voice.

[1735] Input: Audio data

[1736] Output: Emotion data of "tired"

[1737] Step 5:

[1738] (Sensor information integration)

[1739] The server retrieves data from the refrigerator's inventory sensor to check whether pizza ingredients are available, and integrates the sensor information into the analysis results to reinforce the recommendation message.

[1740] Input: Refrigerator inventory information

[1741] Output: Consolidated data

[1742] Step 6:

[1743] (Generating a proposal message)

[1744] The server generates a suggestion message based on the need and emotion of "I want pizza, but I'm tired." Specifically, it creates a message that reads, "It seems you want to eat delicious pizza, but you're tired. Would you like us to order pizza for delivery?"

[1745] Input: Needs data such as "I want to eat pizza," emotion data such as "I'm tired," and sensor information

[1746] Output: Proposal message

[1747] Step 7:

[1748] (Using generative AI models)

[1749] The server provides a prompt to the generative AI model to enhance the suggestion message. An example of a prompt is, "The user said, 'I'm a little tired today... I want to eat some good pizza.' He seems tired. Please create a suggestion message that encourages delivery." An enhanced suggestion message is generated.

[1750] Input: Proposal message, prompt text

[1751] Output: Enhanced suggestion message

[1752] Step 8:

[1753] (Send a proposal message)

[1754] The server sends the enhanced proposal message to the terminal, and the user's terminal displays the proposal message for the user to confirm.

[1755] Input: Enhanced suggestion message

[1756] Output: The suggestion message that is displayed on the user's terminal.

[1757] Step 9:

[1758] (User approval and automatic ordering)

[1759] The user approves the proposed message (for example, by saying "Yes, please place the order"), and the device sends this approval to the server, which then executes the order using the API of the e-commerce site or online sales platform.

[1760] Input: User approval

[1761] Output: Order execution

[1762] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1763] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1764] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1765] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1766] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1767] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1768] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1769] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1770] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1771] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1772] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1773] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1774] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1775] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1776] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1777] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1778] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1779] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1780] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1781] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1782] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1783] The following is further disclosed regarding the above embodiment.

[1784] (Claim 1)

[1785] means for collecting user speech via a voice input device;

[1786] A means for converting the collected voice data into text data;

[1787] A means of analyzing text data using natural language processing (NLP) to determine user needs;

[1788] A means to reinforce the analysis results by linking with other sensor information, and

[1789] means for identifying the required items and generating and transmitting a suggestion message to the user;

[1790] A system that includes a means for placing orders on e-commerce sites and online supermarkets based on user approval.

[1791] (Claim 2)

[1792] 10. The system of claim 1, further comprising means for transmitting the collected voice data to a server in real time.

[1793] (Claim 3)

[1794] The system according to claim 1, further comprising means for analyzing emotions and urgency from users' tweets and conversations and proposing fees based on dynamic pricing.

[1795] "Example 1"

[1796] (Claim 1)

[1797] means for collecting user speech via a voice input device;

[1798] A means for converting the collected voice data into text data;

[1799] A means of analyzing text data using natural language processing (NLP) to determine user needs;

[1800] A means to reinforce the analysis results by linking with other sensor information, and

[1801] means for identifying the required items and generating and transmitting a suggestion message to the user;

[1802] means for executing an order in the electronic commerce system based on the user's authorization;

[1803] means for generating a suggestion message based on the voice data and sensor information and transmitting the message to a user's device;

[1804] The system includes a means for transmitting user authorization information to a server.

[1805] (Claim 2)

[1806] 10. The system of claim 1, further comprising means for transmitting the collected voice data to a server in real time.

[1807] (Claim 3)

[1808] The system according to claim 1, further comprising means for analyzing emotions and urgency from user tweets and conversations and proposing fees based on dynamic pricing.

[1809] "Application Example 1"

[1810] (Claim 1)

[1811] means for collecting user speech via a voice input device;

[1812] A means for converting the collected voice data into text data;

[1813] A means of analyzing text data using natural language processing (NLP) to determine user needs;

[1814] A means to reinforce the analysis results by linking with other sensor information, and

[1815] means for identifying the required items and generating and transmitting a suggestion message to the user;

[1816] A means of placing orders on e-commerce sites and online supermarkets based on user approval;

[1817] A system that includes a means to automatically execute orders using a food delivery API.

[1818] (Claim 2)

[1819] 10. The system of claim 1, further comprising means for transmitting the collected voice data to a server in real time.

[1820] (Claim 3)

[1821] The system according to claim 1, further comprising means for analyzing emotions and urgency from users' tweets and conversations and proposing fees based on dynamic pricing.

[1822] "Example 2: Combining Emotion Engines"

[1823] (Claim 1)

[1824] means for collecting user speech via a voice input device;

[1825] A means for converting the collected voice data into text data;

[1826] a means for analyzing the text data using natural language processing to determine user needs;

[1827] A means to reinforce the analysis results by linking with other sensor information, and

[1828] a means for determining a user's sentiment using a sentiment analysis engine;

[1829] A means for identifying the necessary items and generating and transmitting a suggestion message that takes into account the user's feelings;

[1830] The system includes means for executing an order to an e-commerce platform based on user authorization.

[1831] (Claim 2)

[1832] 10. The system of claim 1, further comprising means for transmitting the collected voice data to a server in real time.

[1833] (Claim 3)

[1834] 10. The system of claim 1, further comprising means for generating the suggestion messages using the generative AI model.

[1835] "Application example 2 when combining emotion engines"

[1836] (Claim 1)

[1837] means for collecting user speech via a voice input device;

[1838] A means for converting the collected voice data into text data;

[1839] A means of analyzing text data using natural language processing (NLP) to determine user needs;

[1840] A means to reinforce the analysis results by linking with other sensor information, and

[1841] means for identifying the required items and generating and transmitting a suggestion message to the user;

[1842] A means for placing an order with an e-commerce site or online sales platform based on the user's authorization;

[1843] a means including an emotion analysis engine for analyzing the user's emotions from the conversation and reflecting the results in the proposed message;

[1844] A means for inputting the suggested message as a prompt into the generative AI model to reinforce the suggested content;

[1845] A system including:

[1846] (Claim 2)

[1847] 10. The system of claim 1, further comprising means for transmitting the collected voice data to a server in real time.

[1848] (Claim 3)

[1849] The system according to claim 1, further comprising means for analyzing emotions and urgency from users' tweets and conversations and proposing fees based on dynamic pricing. [Explanation of symbols]

[1850] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting user speech via a voice input device; A means for converting the collected voice data into text data; A means of analyzing text data using natural language processing (NLP) to determine user needs; A means to reinforce the analysis results by linking with other sensor information, and means for identifying a required item and generating and transmitting a suggestion message to the user; A system that includes a means for placing orders on e-commerce sites and online supermarkets based on user approval.

2. 2. The system of claim 1, further comprising means for transmitting the collected voice data to a server in real time.

3. The system according to claim 1, further comprising means for analyzing emotions and urgency from user tweets and conversations and proposing fees based on dynamic pricing.

Images