system

Abstract

We provide the system. [Solution] Means for collecting user data, A means of analyzing collected user data and learning user behavior patterns and preferences, A means of automatically performing user tasks based on learned behavioral patterns and preferences, A means for sending the results of the executed task to the user's terminal, Means of receiving user feedback, A system that includes means for correcting task results based on received feedback.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern busy lives, many users have difficulty efficiently performing various tasks due to lack of time. In particular, simple tasks such as replying to emails and creating documents consume time, resulting in the problem of being unable to concentrate on important work. There is also a problem of being overwhelmed by too many choices. As a result, users are often troubled by stress related to time and choices. The purpose of this invention is to solve these problems and provide support for users to work efficiently.

Means for Solving the Problems

[0005] To solve the above-mentioned problems, the present invention provides the following system: a system comprising means for collecting user data, means for analyzing the collected user data and learning the user's behavior patterns and preferences, means for automatically executing the user's tasks based on the learned behavior patterns and preferences, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, and means for modifying the task results based on the received feedback. As a result, the user can concentrate on important tasks as the agent performs various tasks on their behalf. Furthermore, the agent continues to learn through feedback, allowing the user to always receive optimal support.

[0006] "User data" refers to data that includes information related to a user's behavior and preferences.

[0007] "Analysis" is the process of examining collected data in detail to find meaning and patterns.

[0008] "Behavioral patterns" are characteristics that indicate the tendencies of a series of actions a user has taken in the past.

[0009] "Preferences" refer to the tendencies of things and behaviors that a user particularly enjoys.

[0010] "Learning" is the process of understanding user behavior patterns and preferences from data using machine learning algorithms.

[0011] "Automatically executing tasks" refers to a function where the system performs various tasks based on user instructions without human intervention.

[0012] "Results" refer to the finished product or output obtained after a task has been completed.

[0013] A "terminal" is a device (such as a smartphone or personal computer) that a user operates to communicate with a system.

[0014] "Feedback" refers to the evaluation or correction instructions given by the user to the system.

[0015] "Correction" means changing the result or process based on feedback.

[0016] "Natural language processing technology" is a technology for the system to understand and analyze human language.

[0017] "Machine learning algorithm" is a technology that automatically learns patterns and knowledge from data.

[0018] "Model" is a mathematical structure constructed by machine learning to reproduce the user's behavior patterns and preferences.

[0019] "Template" is a pre-defined format for arranging information in a specific form.

Brief Explanation of Drawings

[0020] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8]It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0021] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0022] First, the language used in the following description will be explained.

[0023] In the following embodiments, a processor with a reference number (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0024] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0025] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0026] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0028] [First Embodiment]

[0029] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0030] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0033] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0036] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0040] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0041] System Overview

[0042] The present invention is a system that efficiently performs user tasks using the user's personal agent, and specifically includes processes for collecting, analyzing, learning, automatically executing tasks, providing results, receiving feedback, and making corrections based on user data.

[0043] Program Processing Overview

[0044] Learning Module

[0045] The server first collects user data related to the user's behavior and preferences. This includes, for example, the user's past email content, web browsing history, and purchase history. Then, this data is analyzed using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences. This builds a model for predicting the user's future behavior. The model is stored on the server and made available to the agent in real time.

[0046] Task execution module

[0047] The terminal forwards user instructions to the server. For example, if a user instructs the server to "create meeting materials for next week," the server receives the instruction along with the necessary information (date and time, format, and summary of content). The server selects the optimal template based on learned behavioral patterns and preferences and automatically executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are then sent from the server to the terminal.

[0048] Interface module

[0049] The terminal displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part to be described in more detail," that feedback is sent to the server via the terminal. The server analyzes the feedback and makes any necessary corrections. The corrected results are then sent back to the terminal and displayed to the user.

[0050] Explanation with specific examples

[0051] For preparing meeting materials for next week

[0052] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0053] 2. The terminal forwards the instructions to the server, which then selects an appropriate template by referring to past meeting materials and user preferences.

[0054] 3. The server automatically generates meeting materials based on the template.

[0055] 4. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0056] 5. The user reviews the document and enters feedback into the device, such as "I would like this part added."

[0057] 6. The terminal forwards the feedback to the server, which analyzes the feedback and corrects the document.

[0058] 7. The revised document is sent back to the terminal and displayed to the user.

[0059] This frees users from the burden of document creation, allowing them to efficiently focus on important tasks. By having agents automatically perform tasks, users can save significant time and effort.

[0060] The following describes the processing flow.

[0061] Step 1:

[0062] The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0063] Step 2:

[0064] The terminal receives user instructions and transmits those instructions along with necessary information (such as the date, time, format, and summary of the meeting) to the server.

[0065] Step 3:

[0066] The server analyzes the received instructions and selects the most suitable template based on the user's past data (past meeting materials, emails, preference data).

[0067] Step 4:

[0068] The server automatically generates meeting materials by filling in the necessary content based on the selected template.

[0069] Step 5:

[0070] The server transfers the generated meeting materials to the terminal.

[0071] Step 6:

[0072] The terminal displays the meeting materials it has received to the user.

[0073] Step 7:

[0074] The user reviews the displayed meeting materials and enters any necessary feedback into the device, such as "Please elaborate on this section."

[0075] Step 8:

[0076] The device receives user feedback and forwards it to the server.

[0077] Step 9:

[0078] The server analyzes the feedback received and makes revisions to the meeting materials.

[0079] Step 10:

[0080] The server resends the revised meeting materials to the terminal.

[0081] Step 11:

[0082] The terminal displays the revised meeting materials that have been resent to the user.

[0083] Step 12:

[0084] The user performs a final review and provides feedback again if necessary. This cycle continues until the document is finally finalized.

[0085] (Example 1)

[0086] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0087] Traditional task management systems were inefficient because users had to manually set tasks, check the results, and make corrections if necessary. Furthermore, it was difficult to optimize task execution based on individual user preferences and behavioral patterns. Therefore, there was a need for a system that would reduce the burden on users and allow for more efficient task execution.

[0088] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0089] In this invention, the server includes means for collecting user data, means for analyzing the collected user data using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences, means for automatically executing the user's tasks based on the learned behavior patterns and preferences, selecting the optimal template and generating results, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, and means for analyzing the received feedback and modifying the task results. This enables the automation of tasks based on the user's individual preferences and behavior patterns, and adaptive modification based on feedback.

[0090] "User data" refers to information related to a user's behavior and preferences, such as the content of emails, web browsing history, and purchase history.

[0091] "Natural language processing technology" refers to the technology that uses computers to understand, generate, and analyze human language, such as text and audio data.

[0092] "Clustering techniques" refer to analytical methods that group similar data together to understand the structure of the data.

[0093] "Behavioral patterns" refer to the tendencies of a series of actions and operations that a user has performed in the past.

[0094] "Preferences" refer to the individual user's tastes and preferences.

[0095] A "template" refers to a pre-prepared format or template for performing a specific task.

[0096] "Feedback" refers to comments and suggestions for improvements provided by users regarding the results.

[0097] A "machine learning algorithm" refers to a mathematical method used to learn patterns based on data and perform tasks such as prediction and classification.

[0098] Modes for carrying out the invention

[0099] This invention provides a system for collecting, analyzing, learning from, and automatically executing user data, thereby reducing the burden on users and efficiently carrying out tasks. The specific configuration and operation of this system are described below.

[0100] System Configuration

[0101] This system consists of a server for collecting and analyzing user data, and terminals where users input instructions. The server is a high-performance computer system that uses databases, natural language processing technologies (e.g., BERT and GPT-3®), clustering methods (e.g., K-means), and machine learning algorithms. The terminals are devices that users can directly operate (e.g., PCs, smartphones, tablets).

[0102] Program Processing Overview

[0103] The server first collects data related to user behavior and preferences. Specifically, it obtains data through email servers, browser logs, and APIs of online shopping sites. This data includes past email content, web browsing history, and purchase history.

[0104] Next, the server analyzes the collected data using natural language processing techniques (e.g., BERT, GPT-3) and clustering methods (e.g., K-means). Through this analysis, user behavior patterns and preferences are learned, and individual user models are constructed.

[0105] When a user enters a task into the terminal, the terminal forwards this instruction to the server. For example, if a user gives the instruction "Prepare the meeting materials for next week," the server will send the instruction along with the necessary information (date and time, format, and summary of content).

[0106] The server selects the optimal template based on a trained user model and executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are sent from the server to the terminal, which then displays the materials to the user.

[0107] When a user reviews a document and provides feedback as needed, the device forwards this feedback to the server. The server analyzes the feedback and makes any necessary corrections. The corrected document is then sent back to the device and displayed to the user.

[0108] Specific example

[0109] For example, in a task to create meeting materials for the following week, the user inputs the instruction "Create meeting materials for next week" using their device. The device forwards this instruction to the server, which selects an appropriate template by referring to past meeting materials and the user's preferences. Based on this template, the server automatically generates the meeting materials and sends them to the device. When the user reviews the materials and provides feedback such as "I would like this section to be explained in more detail," the server analyzes the feedback and revises the materials. The revised materials are then sent to the device and displayed to the user.

[0110] Example of a prompt

[0111] An example of a prompt message is, "GPT-3, if the user asks you to create meeting materials for next week, generate new materials using past materials as a reference." By using prompts like this, the generative AI model can appropriately perform the task requested by the user.

[0112] As described above, the system of the present invention can automatically perform tasks based on the user's preferences and behavioral patterns, and can make adaptive modifications based on feedback.

[0113] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0114] Step 1: Collecting User Data

[0115] The server collects data related to the user's past behavior and preferences.

[0116] Specific operation: The server connects to the mail server API and retrieves email data from the past six months. It also collects browser browsing history data from log files and retrieves purchase history from shopping sites via API.

[0117] Input: Data from mail servers, browser log files, and shopping site APIs.

[0118] Output: The user's email content, web browsing history, and purchase history are stored in the server's database.

[0119] Step 2: Analyze and learn from user data

[0120] The server analyzes the collected data and learns the user's behavior patterns and preferences.

[0121] Specific operation: The server uses a natural language processing library (e.g., BERT, GPT-3) to extract important keywords from email content. Next, it applies a clustering algorithm (e.g., K-means) to classify web browsing history into different categories.

[0122] Input: User data collected in Step 1.

[0123] Output: The analyzed behavioral patterns and preferences are saved as a user model.

[0124] Step 3: Receiving the Task

[0125] The user enters the task details via their terminal. The task is then transferred to the server.

[0126] Specific action: The user enters "Create meeting materials for next week" into the application's form and presses the submit button. The device sends this data to the server as an HTTP request.

[0127] Input: Task instructions entered by the user on the device.

[0128] Output: The task instruction reaches the server and is added to the processing queue.

[0129] Step 4: Task execution and result generation

[0130] The server executes tasks and generates results based on the user's behavior patterns and preferences.

[0131] Specific operation: The server selects the most suitable meeting material template from user preference data and automatically generates new materials based on past meeting material data. It adds necessary context and content using a generation AI model (e.g., GPT-3).

[0132] Input: User model, past meeting materials, templates.

[0133] Output: The generated meeting materials file is saved to the server.

[0134] Step 5: Providing Results

[0135] The terminal displays the results received from the server to the user.

[0136] Specific operation: The server sends the generated meeting materials file to the terminal, and the terminal's application displays this file in the user interface.

[0137] Input: Meeting materials file sent from the server.

[0138] Output: The user is able to view the meeting materials on their device.

[0139] Step 6: Receiving Feedback

[0140] Users review the materials and provide feedback.

[0141] Specific action: The user enters feedback into a form on their device, stating "I would like this part described in more detail," and presses the submit button. The device then sends this feedback to the server.

[0142] Input: Feedback entered by the user on the device.

[0143] Output: Feedback reaches the server and is analyzed.

[0144] Step 7: Corrections based on feedback

[0145] The server analyzes the feedback it receives and corrects the document.

[0146] Specific operation: The server analyzes the feedback using a natural language processing tool and makes necessary corrections using a generation AI model. The corrected document is then regenerated and sent to the terminal.

[0147] Input: User feedback.

[0148] Output: The revised meeting materials are sent to the terminal and displayed to the user again.

[0149] The above outlines the specific processing steps of the system. This series of steps enables automated task execution and adaptive correction, significantly reducing the burden on the user.

[0150] (Application Example 1)

[0151] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0152] In modern society, the series of tasks and payment management that users perform daily have become increasingly complex, and optimization is needed. In particular, there is a lack of efficient and secure payment methods that suggest the optimal payment method based on the user's purchase history and transaction patterns, which often leads to users wasting time and effort. To address this challenge, a system is needed that collects and analyzes user data and uses trained models to efficiently automate tasks.

[0153] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0154] In this invention, the server includes means for collecting user data, means for analyzing the collected user data and learning the user's behavior patterns and preferences, means for automatically executing the user's tasks based on the learned behavior patterns and preferences, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, means for modifying the task results based on the received feedback, means for analyzing the user's purchase history and transaction patterns and suggesting the optimal payment method, means for generating reminder notifications based on the payment method, and means for quickly processing payments on the user's terminal. This enables efficient and secure task management and payment processing for users, saving them time and effort.

[0155] "User data" refers to various types of information related to a user, such as their purchase history, transaction patterns, and preferences.

[0156] "Analysis" refers to the process of analyzing collected user data to reveal user behavior patterns and preferences.

[0157] "Behavioral patterns" refer to the tendencies and habits of how a user behaves in specific situations.

[0158] "Preferences" refer to the user's tendency to choose things and services that they particularly like.

[0159] A "task" refers to a specific task or operation that a user is required to perform.

[0160] "Device" refers to an electronic device used by a user, and includes smartphones, computers, and other similar devices.

[0161] "Feedback" refers to opinions, evaluations, and requests for corrections provided by users.

[0162] "Modification" refers to the act of changing task results, settings, etc., based on user feedback.

[0163] "Purchase history" refers to information about the products and services that a user has purchased in the past.

[0164] "Trading patterns" refer to the tendencies and habits of a user in their trading.

[0165] "Payment method" refers to the payment method selected by the user, and includes credit cards, debit cards, electronic money, etc.

[0166] A "reminder notification" refers to an automated notification that informs the user of specific tasks, payment deadlines, or other important information.

[0167] "Payment processing" refers to the act of actually completing a monetary transaction using the selected payment method.

[0168] Modes for carrying out the invention

[0169] System Overview

[0170] This invention is a system that efficiently performs user tasks using a user's personal agent. Specifically, it includes processes for collecting, analyzing, and learning user data, automatically executing tasks, providing results, receiving feedback, and making corrections. It also includes analyzing the user's purchase history and transaction patterns, suggesting the optimal payment method, generating payment reminder notifications, and processing them quickly on the terminal.

[0171] Program Processing Overview

[0172] Learning Module

[0173] The server first collects user data related to the user's behavior and preferences. This includes the user's past email content, web browsing history, and purchase history. Next, the collected data is analyzed using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences. Using these learning results, a model is built to predict the user's future behavior. The model is stored on the server and made available to the agent in real time.

[0174] Task execution module

[0175] The terminal forwards user instructions to the server. For example, if a user instructs the server to "create meeting materials for next week," the server receives the instruction along with the necessary information (date and time, format, and summary of content). The server selects the optimal template based on learned behavioral patterns and preferences and automatically executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are then sent from the server to the terminal.

[0176] Interface module

[0177] The terminal displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user enters feedback such as "I would like this part described in more detail" into the terminal, that feedback is sent to the server. The server analyzes the feedback, makes any necessary corrections, and sends the corrected results back to the terminal.

[0178] Payment management module

[0179] The server collects and analyzes the user's purchase history and transaction patterns to suggest the most suitable payment method. For example, it suggests the best payment method from options such as credit cards, debit cards, and electronic money based on the user's past usage. Furthermore, if a payment deadline is approaching, it generates a reminder notification and automatically sends it to the user's device.

[0180] Specific example

[0181] For preparing meeting materials for next week

[0182] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0183] 2. The terminal forwards the instructions to the server, which then selects an appropriate template by referring to past meeting materials and user preferences.

[0184] 3. The server automatically generates meeting materials based on the template.

[0185] 4. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0186] 5. The user reviews the document and enters feedback into the device, such as "I would like this part added."

[0187] 6. The terminal forwards the feedback to the server, which analyzes the feedback and corrects the document.

[0188] 7. The revised document is sent back to the terminal and displayed to the user.

[0189] In the case of payment management

[0190] 1. The server collects and analyzes the user's purchase history and transaction patterns.

[0191] 2. Based on the analysis results, the server suggests the most suitable payment method to the user.

[0192] 3. The user selects a suggested payment method and completes the payment on the terminal.

[0193] 4. When the payment deadline approaches, the server generates a reminder notification and sends it to the user's device.

[0194] Example of a prompt

[0195] Suggest the best payment method for user ID 12345, based on their first-quarter purchase history data.

[0196] This system frees users from the burden of creating documents and allows them to efficiently and securely select and manage payment methods.

[0197] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0198] Step 1:

[0199] The server collects user data (purchase history, transaction patterns, preferences, etc.). Specifically, it retrieves information such as the user's past email content, web browsing history, and purchase history from a database. This allows for the systematic accumulation of user data.

[0200] Step 2:

[0201] The server analyzes the collected user data. It uses natural language processing techniques to analyze the data and learn user behavior patterns and preferences. Specifically, it tokenizes text data and performs sentiment analysis and relationship extraction. It uses clustering techniques to classify the data into groups. This allows it to build a model for predicting future user behavior.

[0202] Step 3:

[0203] The server automatically performs tasks based on the learning results. Specifically, when a user gives an instruction such as "Create meeting materials for next week," the necessary information (date and time, format, and summary of content) is entered along with the instruction. The server refers to the learned model, automatically selects the optimal template, and generates the meeting materials using historical data.

[0204] Step 4:

[0205] The server sends the results of the executed task to the terminal. The generated meeting materials are sent from the server to the terminal and displayed to the user. This allows the user to review the generated materials.

[0206] Step 5:

[0207] Users review the generated documents and provide feedback. They input feedback, such as "I'd like this section described in more detail," through their device, and this feedback information is sent to the server.

[0208] Step 6:

[0209] The server analyzes the received feedback and corrects the task results. Specifically, based on the feedback, it uses natural language processing techniques to modify and add to the text, updating the document. A newly corrected document is then generated.

[0210] Step 7:

[0211] The revised document is sent again from the server to the terminal and displayed to the user. The user can then review the revised document.

[0212] Step 8:

[0213] The server analyzes the user's purchase history and transaction patterns to suggest the optimal payment method. Specifically, it uses clustering techniques and other machine learning algorithms to determine the best payment method (credit card, debit card, e-money, etc.) for the user.

[0214] Step 9:

[0215] The server sends payment methods to the user's terminal and displays them to the user. The user reviews the suggestions and selects a payment method.

[0216] Step 10:

[0217] The payment is processed on the terminal based on the payment method selected by the user. Specifically, the payment process is initiated and the payment is completed.

[0218] Step 11:

[0219] As the payment deadline approaches, the server automatically generates a reminder notification and sends it to the user's device. This helps users remember to make their payments.

[0220] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0221] System Overview

[0222] The present invention provides a system that efficiently performs user tasks using a user's personal agent, and in particular, offers more personalized assistance by combining it with an emotion engine that recognizes the user's emotions. The system includes processes for collecting, analyzing, learning from, automatically executing tasks, providing results, receiving feedback, and making corrections to user data.

[0223] Program Processing Overview

[0224] Introducing an emotional engine

[0225] The server collects data related to user behavior and preferences, as well as data to recognize the user's emotional state. Specifically, it analyzes emotions from the user's text, voice, and facial expressions, and uses an emotion engine to recognize this data as emotions. For example, it analyzes the user's facial expressions and tone of voice while they are reading an email to determine their stress level, joy, sadness, etc.

[0226] Learning Module

[0227] The server analyzes the collected user data and sentiment data. This allows it to learn user behavior patterns, preferences, and emotional states. Natural language processing techniques, clustering methods, and sentiment recognition algorithms are used in the analysis. Based on the obtained information, a model is built to address the user's emotional state. This model is stored on the server and made available to the agent in real time.

[0228] Task execution module

[0229] The terminal forwards user instructions to the server. If the user instructs, "Prepare meeting materials for next week," the server receives the instruction along with necessary information (date, time, format, summary of content) and the user's current emotional state. Based on learned behavioral patterns, preferences, and emotional states, the server selects the most suitable template and automatically executes the task. For example, if the user is stressed, the server may generate more concise and simpler materials. The generated results are then sent from the server back to the terminal.

[0230] Interface module

[0231] The device displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part to be described in more detail," that feedback is sent to the server via the device. The server analyzes the feedback and the user's emotional state and makes any necessary corrections. The corrected results are then sent back to the device and displayed to the user.

[0232] Explanation with specific examples

[0233] For preparing meeting materials for next week

[0234] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0235] 2. The device transmits instructions, necessary information, and user sentiment data to the server.

[0236] 3. The server references the user's past data and emotional state and selects an appropriate template.

[0237] 4. The server automatically generates meeting materials based on templates. For example, if a user is feeling stressed, it will generate a document that summarizes the information concisely.

[0238] 5. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0239] 6. Users review the materials and provide feedback as needed. For example, they might enter feedback such as, "I would like this section to be explained in more detail."

[0240] 7. The device transfers feedback and sentiment data to the server.

[0241] 8. The server analyzes feedback and sentiment data and modifies the document.

[0242] 9. The revised document is sent back to the terminal and displayed to the user.

[0243] This system not only frees users from the burden of document creation but also allows them to receive optimal support tailored to their emotional state. Agents automatically perform tasks and respond to the user's emotional state, significantly saving time and effort.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0247] Step 2:

[0248] The device receives user instructions and simultaneously collects user emotional data (text, voice, facial expressions, etc.).

[0249] Step 3:

[0250] The device analyzes the user's emotional data and uses an emotion engine to recognize the user's current emotional state.

[0251] Step 4:

[0252] The terminal transmits the instructions, necessary information (date, time, format, and summary of the meeting), and emotional state to the server.

[0253] Step 5:

[0254] The server analyzes the instructions and emotional state received, and selects the optimal template based on the user's past data (past meeting materials, emails, preference data).

[0255] Step 6:

[0256] The server automatically fills in the necessary content based on the selected template and generates meeting materials. For example, if the user is under stress, it will create materials that are concise and easy to understand visually.

[0257] Step 7:

[0258] The server transfers the generated meeting materials to the terminal.

[0259] Step 8:

[0260] The terminal displays the meeting materials it has received to the user.

[0261] Step 9:

[0262] The user reviews the displayed meeting materials and enters any necessary feedback into the device, such as "Please elaborate on this section."

[0263] Step 10:

[0264] The device then collects user feedback and their emotional state at that time, and transmits it to the server.

[0265] Step 11:

[0266] The server analyzes the feedback and emotional state received and revises the meeting materials accordingly. For example, if the user is in an optimistic emotional state, it creates more comprehensive materials that include additional details.

[0267] Step 12:

[0268] The server resends the revised meeting materials to the terminal.

[0269] Step 13:

[0270] The terminal displays the revised meeting materials that have been resent to the user.

[0271] Step 14:

[0272] The user performs a final review and provides feedback again if necessary. This cycle continues until the document is finally finalized.

[0273] (Example 2)

[0274] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0275] Traditional personal agents could automate tasks based on user behavior patterns and preferences, but they could not consider the user's emotional state. As a result, they could not provide appropriate support depending on the user's emotional state, and in some cases, this could increase the user's stress. Furthermore, the automated execution of tasks and the incorporation of feedback often did not yield optimal results because they could not consider the emotional state. Therefore, there is a need for improved user satisfaction and efficient task execution.

[0276] In Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting user data, means for recognizing the user's emotional state using an emotion engine, means for analyzing the collected user data and emotional data to learn the user's behavioral patterns, preferences, and emotional state, means for automatically executing the user's tasks based on the learned behavioral patterns, preferences, and emotional state, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, and means for modifying the task results based on the received feedback and emotional data. This enables personalized support that corresponds to the user's emotional state, allowing for efficient task execution and improved user satisfaction.

[0277] "User data" refers to information such as user behavior, preferences, and instructions.

[0278] "Emotional data" refers to information related to the emotional state extracted from the user's expressions, voice tones, and texts.

[0279] "Emotion engine" refers to the technology or software for analyzing and recognizing the user's emotional state.

[0280] "Natural language processing technology" refers to the technology for analyzing text data and understanding its meaning.

[0281] "Clustering method" refers to the technology for grouping data according to similar characteristics.

[0282] "Generative AI model" refers to the model for generating text and other data using artificial intelligence.

[0283] "Machine learning algorithm" refers to the algorithm for learning from experience and recognizing patterns.

[0284] "User's behavior pattern" refers to the behavior tendency of the user under specific circumstances.

[0285] "Preference" refers to what the user likes or the tendency.

[0286] "Feedback" refers to the opinions or correction requests provided by the user.

[0287] This invention is a system that efficiently performs the user's tasks using the user's personal agent, and in particular, provides more personalized support by combining an emotion engine that recognizes the user's emotions. The specific embodiments of this system will be described below.

[0288] Introduction of Emotion Engine

[0289] The server collects data related to user behavior and preferences, as well as data to recognize the user's emotional state. Specifically, it analyzes emotions from the user's text, voice, and facial expressions, and uses an emotion engine to recognize this data as emotions. For example, it analyzes the user's facial expressions and tone of voice while they are reading an email to determine their stress level, joy, sadness, etc.

[0290] Learning Module

[0291] The server analyzes the collected user data and sentiment data. This allows it to learn user behavior patterns, preferences, and emotional states. Natural language processing techniques, clustering methods, and sentiment recognition algorithms are used for the analysis. Based on the obtained information, a model is built to respond to the user's emotional state. This model is stored on the server and made available to the agent in real time.

[0292] Task execution module

[0293] The terminal forwards user instructions to the server. For example, if a user instructs the terminal to "create meeting materials for next week," the server receives the instruction along with necessary information (date and time, format, summary of content) and the user's current emotional state. The server selects the most suitable template based on learned behavioral patterns, preferences, and emotional states, and automatically executes the task. For example, if the user is stressed, the server may generate more concise and simpler materials. The generated results are then sent from the server back to the terminal.

[0294] Interface module

[0295] The device displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part described in more detail," that feedback is sent to the server via the device. The server analyzes the feedback and the user's emotional state and makes any necessary corrections. The corrected results are then sent back to the device and displayed to the user.

[0296] Specific example

[0297] Next, let's consider a specific example: preparing meeting materials for next week.

[0298] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0299] 2. The device transmits instructions, necessary information, and user sentiment data to the server.

[0300] 3. The server references the user's past data and emotional state and selects an appropriate template.

[0301] 4. The server automatically generates meeting materials based on templates. For example, if a user is feeling stressed, it will generate a document that summarizes the information concisely.

[0302] 5. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0303] 6. The user reviews the document and provides feedback as needed. For example, they might enter feedback such as, "I would like this section to be explained in more detail."

[0304] 7. The device transfers feedback and sentiment data to the server.

[0305] 8. The server revises the document based on the feedback and sends it back to the terminal. The final result is displayed to the user.

[0306] With this system, users can efficiently perform tasks and receive optimal support according to their emotions.

[0307] Examples of prompt sentences

[0308] "Please create the meeting materials for next week. Please summarize the information concisely considering your stress level."

[0309] The flow of the specific process in Example 2 will be described using FIG. 13.

[0310] Step 1:

[0311] The user uses the terminal to input an instruction such as "Create the meeting materials for next week." This input includes the instruction content, necessary related information (e.g., date and time, format, summary of content). The terminal recognizes this and further determines the user's current emotional state (collected from expressions, voice, etc.). As input data, the instruction text and emotional data are acquired. This data is temporarily stored in the terminal.

[0312] Step 2:

[0313] The terminal transfers the collected instruction content and emotional data to the server. The input includes the instruction text, date and time, format, summary of content, and emotional data. The output after transfer is realized when the server receives these data. As a specific operation, the terminal uses network communication to send the data to the server.

[0314] Step 3:

[0315] The server analyzes the received instruction text and sentiment data. Natural language processing techniques are used for the analysis. The input data consists of instruction text and sentiment data. The analysis results in an understanding of the instruction content and a determination of the emotional state. The output includes the analysis results and the user's emotional state. Specifically, the server executes a text analysis engine and an sentiment recognition algorithm.

[0316] Step 4:

[0317] The server selects an appropriate template based on a model that has learned past behavioral patterns, preferences, and emotional states. Input data includes analysis results and historical data. This data is evaluated using clustering techniques to determine the optimal template. The output is the selected template. Specifically, the server retrieves historical data from the database, applies the machine learning model, and selects a template.

[0318] Step 5:

[0319] The server creates documents using the selected template. A generative AI model is used here. The input data includes the template and the necessary information. Since the generated documents also take emotional states into account, for example, a concise document will be generated for a user experiencing stress. The output is the generated document. Specifically, the server runs the generative AI model and embeds information into the template.

[0320] Step 6:

[0321] The server sends the generated document to the terminal. The data sent is the generated document. Specifically, the server sends the data to the terminal using network communication. The output is the terminal receiving the document.

[0322] Step 7:

[0323] The terminal displays the received materials to the user. The input data is generated material sent from the server. Specifically, the terminal uses its screen display function to visually provide the materials to the user. The output is that the materials become viewable by the user.

[0324] Step 8:

[0325] The user reviews the document and enters feedback. The input data consists of user correction instructions and additional information. Specifically, the user enters text using a terminal. The output is the feedback text.

[0326] Step 9:

[0327] The terminal retransmits the input feedback and sentiment data to the server. The input data consists of feedback text and sentiment data. Specifically, the terminal sends data to the server using network communication. The output is the server receiving this data again.

[0328] Step 10:

[0329] The server analyzes user feedback and sentiment data to revise the document. Input data consists of feedback text and sentiment data. The analysis determines the necessary revisions to the document. The output is the revised document. Specifically, the server runs the AI ​​model again to update the document.

[0330] Step 11:

[0331] The server resends the corrected document to the terminal. The input data is the corrected document. Specifically, the server sends the data to the terminal using network communication. The output is the terminal receiving the corrected document.

[0332] Step 12:

[0333] The terminal redisplays the revised document to the user. The input data is the revised document. Specifically, the terminal uses its screen display function to visually present the document to the user again. The output is that the revised document is made viewable by the user.

[0334] In this way, the system can efficiently perform tasks while taking the user's emotional state into consideration.

[0335] (Application Example 2)

[0336] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0337] Traditional personal agent systems provided the functionality to automate tasks based on user behavior patterns and preferences, but they did not take user emotions into consideration. As a result, users may experience stress or dissatisfaction. Especially in physical stores, responding to customers' emotional states is crucial, but this was difficult with current systems. Therefore, there was a need for technology that could provide more appropriate and personalized customer service based on customer emotions.

[0338] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0339] In this invention, the server includes means for collecting user data, means for analyzing the collected user data and emotional data to learn the user's behavioral patterns, preferences, and emotional states, and means for automatically executing user tasks based on the learned behavioral patterns, preferences, and emotional states. This makes it possible to detect the customer's emotional state in real time and provide optimal advice accordingly.

[0340] "User data" refers to all information about a user, including their behavior, preferences, instructions, and feedback.

[0341] "Emotional data" refers to information about a user's emotional state, analyzed from their facial expressions, voice, text, etc.

[0342] "Analysis" refers to the process of evaluating and classifying collected data to extract meaningful information.

[0343] "Behavioral patterns" refer to the tendencies of actions and habits that users repeatedly perform.

[0344] "Preferences" refer to a user's likes and dislikes, or their tendencies to prefer certain things.

[0345] "Learning" refers to the process where an algorithm uses data to build a model of user behavior patterns and preferences, and then uses that model for prediction and decision-making.

[0346] A "task" refers to a series of actions or procedures performed based on instructions from a user.

[0347] "Execute automatically" means that the system performs a pre-configured process without user intervention.

[0348] "Terminal" refers to information processing devices such as computers, smartphones, and tablets used by users.

[0349] "Feedback" refers to evaluations and opinions provided by users regarding the results.

[0350] "Correction" refers to making improvements or changes to the initial results based on the feedback provided.

[0351] "Detecting customer emotional states in real time" means recognizing customer emotions instantly, without any time lag.

[0352] "Advice" refers to the recommended actions or instructions provided by the system.

[0353] This invention realizes a system that combines an emotion engine with a user's personal agent to analyze the emotional state of the user and customer and provide optimal advice accordingly. Each part of the system operates in response to the server, terminal, and user.

[0354] System Overview

[0355] The server has the capability to collect and analyze user data and sentiment data. This utilizes natural language processing techniques and sentiment recognition algorithms. Specifically, software such as TENSORFLOW®, OpenCV, and SpeechRecognition are used. The collected data is used to learn the user's behavioral patterns, preferences, and emotional states using machine learning algorithms. The server also automatically performs user tasks based on the learned model and sends the results to the user's terminal.

[0356] The device displays results sent from the server to the user and receives real-time feedback. It also plays a role in transferring the user's emotional data to the server. Smart glasses are equipped with a camera and microphone, which can capture the user's (or customer's) facial expressions and voice. This allows for real-time analysis of customer emotions and provides staff with the most appropriate advice.

[0357] Program Processing Overview

[0358] The server collects and analyzes user data and identifies emotional data using an emotion engine. For example, it can analyze emotions from a user's text, voice, and facial expressions, and recognize their emotional state in real time. The server also uses machine learning algorithms to learn user behavior patterns and preferences from the collected data. Based on this, it automatically executes tasks and sends the results to the terminal.

[0359] The terminal displays the results sent from the server to the user and receives user feedback. The feedback is sent to the server for further analysis and correction. The corrected results are then sent back to the terminal and displayed to the user.

[0360] As a concrete example, consider the following scenario: While a customer is examining a product, a staff member wears smart glasses. The system analyzes the customer's facial expression, and if it detects a satisfied expression, the staff member's glasses display the advice, "The customer is happy. Let's suggest additional products." In another scenario, if the customer appears confused, the system provides the advice, "The customer is confused. Let's provide a more detailed explanation."

[0361] An example of a specific prompt for a generative AI model is: "Please provide a prompt for a generative AI model to recognize a customer's smile and advise on how to respond if the customer is happy." This will enable the AI ​​model to automatically generate appropriate advice.

[0362] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0363] Step 1:

[0364] The server collects user and sentiment data. Specifically, it collects data such as user and customer behavior, preferences, facial expressions, voice, and text. This input data is sent to the server and stored in a database. This data collection provides initial information for use throughout the system.

[0365] Step 2:

[0366] The server analyzes the collected user data and emotion data. Natural language processing techniques and emotion recognition algorithms are used in this process. Specifically, machine learning libraries such as Python's TensorFlow are used to analyze text data, and OpenCV is used to analyze facial expression data. The analysis results are extracted as user behavior patterns, preferences, and emotional states.

[0367] Step 3:

[0368] Based on the analysis results, the server uses machine learning algorithms to model and learn user behavior patterns, preferences, and emotional states. This generates a predictive model for future task performance. The model's accuracy improves through sequential feedback during this learning process.

[0369] Step 4:

[0370] The user or device instructs the server to perform a specific task. For example, it might instruct the server to "prepare the meeting materials for next week." This instruction includes information such as the date and time, format, and a summary of the content, as well as the user's current emotional state. This becomes the input sent to the server.

[0371] Step 5:

[0372] The server automatically performs tasks requested by the user based on a trained model. For example, if the user is stressed, it automatically generates simple and easy-to-understand meeting materials. In this process, it uses a generative AI model to select and process appropriate templates and content, and then outputs the final materials.

[0373] Step 6:

[0374] The server sends the task results to the terminal. The generated materials and advice are delivered to the user's terminal and displayed to the user. At this point, output to the user is complete, and the user can review the results as needed.

[0375] Step 7:

[0376] Users provide feedback on the results displayed on their devices. For example, they can send detailed requests to the server via their devices, such as "Please describe this part in more detail." This becomes the user's input data.

[0377] Step 8:

[0378] The server analyzes the received feedback and makes necessary corrections. Specifically, it uses machine learning algorithms again to adjust the task results, taking the feedback into consideration, and makes appropriate corrections. The corrected results are then sent back to the terminal.

[0379] Step 9:

[0380] The device then redisplays the corrected results to the user. This is where final confirmation takes place, and the user can provide further feedback on the results. This ensures that the optimal result, tailored to the user's requirements down to the smallest detail, is achieved.

[0381] Specific actions

[0382] The smart glasses capture the customer's facial expressions and voice in real time using their camera and microphone.

[0383] The acquired data is sent to a server and analyzed using natural language processing technology and emotion recognition algorithms.

[0384] If the customer is happy, advise the staff, "The customer is happy. Let's suggest additional products."

[0385] If the customer is confused, advise them to "The customer is confused. Let's provide a detailed explanation."

[0386] An example of a specific prompt for a generative AI model is: "Please provide a prompt for a generative AI model to recognize a customer's smile and advise on how to respond if the customer is happy." This will enable the AI ​​model to automatically generate appropriate advice.

[0387] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0388] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0389] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0390] [Second Embodiment]

[0391] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0392] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0393] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0394] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0395] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0396] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0397] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0398] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0399] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0400] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0401] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0402] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0403] System Overview

[0404] The present invention is a system that efficiently performs user tasks using the user's personal agent, and specifically includes processes for collecting, analyzing, learning, automatically executing tasks, providing results, receiving feedback, and making corrections based on user data.

[0405] Program Processing Overview

[0406] Learning Module

[0407] The server first collects user data related to the user's behavior and preferences. This includes, for example, the user's past email content, web browsing history, and purchase history. Then, this data is analyzed using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences. This builds a model for predicting the user's future behavior. The model is stored on the server and made available to the agent in real time.

[0408] Task execution module

[0409] The terminal forwards user instructions to the server. For example, if a user instructs the server to "create meeting materials for next week," the server receives the instruction along with the necessary information (date and time, format, and summary of content). The server selects the optimal template based on learned behavioral patterns and preferences and automatically executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are then sent from the server to the terminal.

[0410] Interface module

[0411] The terminal displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part to be described in more detail," that feedback is sent to the server via the terminal. The server analyzes the feedback and makes any necessary corrections. The corrected results are then sent back to the terminal and displayed to the user.

[0412] Explanation with specific examples

[0413] For preparing meeting materials for next week

[0414] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0415] 2. The terminal forwards the instructions to the server, which then selects an appropriate template by referring to past meeting materials and user preferences.

[0416] 3. The server automatically generates meeting materials based on the template.

[0417] 4. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0418] 5. The user reviews the document and enters feedback into the device, such as "I would like this part added."

[0419] 6. The terminal forwards the feedback to the server, which analyzes the feedback and corrects the document.

[0420] 7. The revised document is sent back to the terminal and displayed to the user.

[0421] This frees users from the burden of document creation, allowing them to efficiently focus on important tasks. By having agents automatically perform tasks, users can save significant time and effort.

[0422] The following describes the processing flow.

[0423] Step 1:

[0424] The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0425] Step 2:

[0426] The terminal receives user instructions and transmits those instructions along with necessary information (such as the date, time, format, and summary of the meeting) to the server.

[0427] Step 3:

[0428] The server analyzes the received instructions and selects the most suitable template based on the user's past data (past meeting materials, emails, preference data).

[0429] Step 4:

[0430] The server automatically generates meeting materials by filling in the necessary content based on the selected template.

[0431] Step 5:

[0432] The server transfers the generated meeting materials to the terminal.

[0433] Step 6:

[0434] The terminal displays the meeting materials it has received to the user.

[0435] Step 7:

[0436] The user reviews the displayed meeting materials and enters any necessary feedback into the device, such as "Please elaborate on this section."

[0437] Step 8:

[0438] The device receives user feedback and forwards it to the server.

[0439] Step 9:

[0440] The server analyzes the feedback received and makes revisions to the meeting materials.

[0441] Step 10:

[0442] The server resends the revised meeting materials to the terminal.

[0443] Step 11:

[0444] The terminal displays the revised meeting materials that have been resent to the user.

[0445] Step 12:

[0446] The user performs a final review and provides feedback again if necessary. This cycle continues until the document is finally finalized.

[0447] (Example 1)

[0448] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0449] Traditional task management systems were inefficient because users had to manually set tasks, check the results, and make corrections if necessary. Furthermore, it was difficult to optimize task execution based on individual user preferences and behavioral patterns. Therefore, there was a need for a system that would reduce the burden on users and allow for more efficient task execution.

[0450] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0451] In this invention, the server includes means for collecting user data, means for analyzing the collected user data using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences, means for automatically executing the user's tasks based on the learned behavior patterns and preferences, selecting the optimal template and generating results, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, and means for analyzing the received feedback and modifying the task results. This enables the automation of tasks based on the user's individual preferences and behavior patterns, and adaptive modification based on feedback.

[0452] "User data" refers to information related to a user's behavior and preferences, such as the content of emails, web browsing history, and purchase history.

[0453] "Natural language processing technology" refers to the technology that uses computers to understand, generate, and analyze human language, such as text and audio data.

[0454] "Clustering techniques" refer to analytical methods that group similar data together to understand the structure of the data.

[0455] "Behavioral patterns" refer to the tendencies of a series of actions and operations that a user has performed in the past.

[0456] "Preferences" refer to the individual user's tastes and preferences.

[0457] A "template" refers to a pre-prepared format or template for performing a specific task.

[0458] "Feedback" refers to comments and suggestions for improvements provided by users regarding the results.

[0459] A "machine learning algorithm" refers to a mathematical method used to learn patterns based on data and perform tasks such as prediction and classification.

[0460] Modes for carrying out the invention

[0461] This invention provides a system for collecting, analyzing, learning from, and automatically executing user data, thereby reducing the burden on users and efficiently carrying out tasks. The specific configuration and operation of this system are described below.

[0462] System Configuration

[0463] This system consists of a server for collecting and analyzing user data, and terminals where users input instructions. The server is a high-performance computer system that uses databases, natural language processing techniques (e.g., BERT and GPT-3), clustering methods (e.g., K-means), and machine learning algorithms. The terminals are devices that users can directly operate (e.g., PCs, smartphones, tablets).

[0464] Program Processing Overview

[0465] The server first collects data related to user behavior and preferences. Specifically, it obtains data through email servers, browser logs, and APIs of online shopping sites. This data includes past email content, web browsing history, and purchase history.

[0466] Next, the server analyzes the collected data using natural language processing techniques (e.g., BERT, GPT-3) and clustering methods (e.g., K-means). Through this analysis, user behavior patterns and preferences are learned, and individual user models are constructed.

[0467] When a user enters a task into the terminal, the terminal forwards this instruction to the server. For example, if a user gives the instruction "Prepare the meeting materials for next week," the server will send the instruction along with the necessary information (date and time, format, and summary of content).

[0468] The server selects the optimal template based on a trained user model and executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are sent from the server to the terminal, which then displays the materials to the user.

[0469] When a user reviews a document and provides feedback as needed, the device forwards this feedback to the server. The server analyzes the feedback and makes any necessary corrections. The corrected document is then sent back to the device and displayed to the user.

[0470] Specific example

[0471] For example, in a task to create meeting materials for the following week, the user inputs the instruction "Create meeting materials for next week" using their device. The device forwards this instruction to the server, which selects an appropriate template by referring to past meeting materials and the user's preferences. Based on this template, the server automatically generates the meeting materials and sends them to the device. When the user reviews the materials and provides feedback such as "I would like this section to be explained in more detail," the server analyzes the feedback and revises the materials. The revised materials are then sent to the device and displayed to the user.

[0472] Example of a prompt

[0473] An example of a prompt message is, "GPT-3, if the user asks you to create meeting materials for next week, generate new materials using past materials as a reference." By using prompts like this, the generative AI model can appropriately perform the task requested by the user.

[0474] As described above, the system of the present invention can automatically perform tasks based on the user's preferences and behavioral patterns, and can make adaptive modifications based on feedback.

[0475] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0476] Step 1: Collecting User Data

[0477] The server collects data related to the user's past behavior and preferences.

[0478] Specific operation: The server connects to the mail server API and retrieves email data from the past six months. It also collects browser browsing history data from log files and retrieves purchase history from shopping sites via API.

[0479] Input: Data from mail servers, browser log files, and shopping site APIs.

[0480] Output: The user's email content, web browsing history, and purchase history are stored in the server's database.

[0481] Step 2: Analyze and learn from user data

[0482] The server analyzes the collected data and learns the user's behavior patterns and preferences.

[0483] Specific operation: The server uses a natural language processing library (e.g., BERT, GPT-3) to extract important keywords from email content. Next, it applies a clustering algorithm (e.g., K-means) to classify web browsing history into different categories.

[0484] Input: User data collected in Step 1.

[0485] Output: The analyzed behavioral patterns and preferences are saved as a user model.

[0486] Step 3: Receiving the Task

[0487] The user enters the task details via their terminal. The task is then transferred to the server.

[0488] Specific action: The user enters "Create meeting materials for next week" into the application's form and presses the submit button. The device sends this data to the server as an HTTP request.

[0489] Input: Task instructions entered by the user on the device.

[0490] Output: The task instruction reaches the server and is added to the processing queue.

[0491] Step 4: Task execution and result generation

[0492] The server executes tasks and generates results based on the user's behavior patterns and preferences.

[0493] Specific operation: The server selects the most suitable meeting material template from user preference data and automatically generates new materials based on past meeting material data. It adds necessary context and content using a generation AI model (e.g., GPT-3).

[0494] Input: User model, past meeting materials, templates.

[0495] Output: The generated meeting materials file is saved to the server.

[0496] Step 5: Providing Results

[0497] The terminal displays the results received from the server to the user.

[0498] Specific operation: The server sends the generated meeting materials file to the terminal, and the terminal's application displays this file in the user interface.

[0499] Input: Meeting materials file sent from the server.

[0500] Output: The user is able to view the meeting materials on their device.

[0501] Step 6: Receiving Feedback

[0502] Users review the materials and provide feedback.

[0503] Specific action: The user enters feedback into a form on their device, stating "I would like this part described in more detail," and presses the submit button. The device then sends this feedback to the server.

[0504] Input: Feedback entered by the user on the device.

[0505] Output: Feedback reaches the server and is analyzed.

[0506] Step 7: Corrections based on feedback

[0507] The server analyzes the feedback it receives and corrects the document.

[0508] Specific operation: The server analyzes the feedback using a natural language processing tool and makes necessary corrections using a generation AI model. The corrected document is then regenerated and sent to the terminal.

[0509] Input: User feedback.

[0510] Output: The revised meeting materials are sent to the terminal and displayed to the user again.

[0511] The above outlines the specific processing steps of the system. This series of steps enables automated task execution and adaptive correction, significantly reducing the burden on the user.

[0512] (Application Example 1)

[0513] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0514] In modern society, the series of tasks and payment management that users perform daily have become increasingly complex, and optimization is needed. In particular, there is a lack of efficient and secure payment methods that suggest the optimal payment method based on the user's purchase history and transaction patterns, which often leads to users wasting time and effort. To address this challenge, a system is needed that collects and analyzes user data and uses trained models to efficiently automate tasks.

[0515] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0516] In this invention, the server includes means for collecting user data, means for analyzing the collected user data and learning the user's behavior patterns and preferences, means for automatically executing the user's tasks based on the learned behavior patterns and preferences, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, means for modifying the task results based on the received feedback, means for analyzing the user's purchase history and transaction patterns and suggesting the optimal payment method, means for generating reminder notifications based on the payment method, and means for quickly processing payments on the user's terminal. This enables efficient and secure task management and payment processing for users, saving them time and effort.

[0517] "User data" refers to various types of information related to a user, such as their purchase history, transaction patterns, and preferences.

[0518] "Analysis" refers to the process of analyzing collected user data to reveal user behavior patterns and preferences.

[0519] "Behavioral patterns" refer to the tendencies and habits of how a user behaves in specific situations.

[0520] "Preferences" refer to the user's tendency to choose things and services that they particularly like.

[0521] A "task" refers to a specific task or operation that a user is required to perform.

[0522] "Device" refers to an electronic device used by a user, and includes smartphones, computers, and other similar devices.

[0523] "Feedback" refers to opinions, evaluations, and requests for corrections provided by users.

[0524] "Modification" refers to the act of changing task results, settings, etc., based on user feedback.

[0525] "Purchase history" refers to information about the products and services that a user has purchased in the past.

[0526] "Trading patterns" refer to the tendencies and habits of a user in their trading.

[0527] "Payment method" refers to the payment method selected by the user, and includes credit cards, debit cards, electronic money, etc.

[0528] A "reminder notification" refers to an automated notification that informs the user of specific tasks, payment deadlines, or other important information.

[0529] "Payment processing" refers to the act of actually completing a monetary transaction using the selected payment method.

[0530] Modes for carrying out the invention

[0531] System Overview

[0532] This invention is a system that efficiently performs user tasks using a user's personal agent. Specifically, it includes processes for collecting, analyzing, and learning user data, automatically executing tasks, providing results, receiving feedback, and making corrections. It also includes analyzing the user's purchase history and transaction patterns, suggesting the optimal payment method, generating payment reminder notifications, and processing them quickly on the terminal.

[0533] Program Processing Overview

[0534] Learning Module

[0535] The server first collects user data related to the user's behavior and preferences. This includes the user's past email content, web browsing history, and purchase history. Next, the collected data is analyzed using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences. Using these learning results, a model is built to predict the user's future behavior. The model is stored on the server and made available to the agent in real time.

[0536] Task execution module

[0537] The terminal forwards user instructions to the server. For example, if a user instructs the server to "create meeting materials for next week," the server receives the instruction along with the necessary information (date and time, format, and summary of content). The server selects the optimal template based on learned behavioral patterns and preferences and automatically executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are then sent from the server to the terminal.

[0538] Interface module

[0539] The terminal displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user enters feedback such as "I would like this part described in more detail" into the terminal, that feedback is sent to the server. The server analyzes the feedback, makes any necessary corrections, and sends the corrected results back to the terminal.

[0540] Payment management module

[0541] The server collects and analyzes the user's purchase history and transaction patterns to suggest the most suitable payment method. For example, it suggests the best payment method from options such as credit cards, debit cards, and electronic money based on the user's past usage. Furthermore, if a payment deadline is approaching, it generates a reminder notification and automatically sends it to the user's device.

[0542] Specific example

[0543] For preparing meeting materials for next week

[0544] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0545] 2. The terminal forwards the instructions to the server, which then selects an appropriate template by referring to past meeting materials and user preferences.

[0546] 3. The server automatically generates meeting materials based on the template.

[0547] 4. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0548] 5. The user reviews the document and enters feedback into the device, such as "I would like this part added."

[0549] 6. The terminal forwards the feedback to the server, which analyzes the feedback and corrects the document.

[0550] 7. The revised document is sent back to the terminal and displayed to the user.

[0551] In the case of payment management

[0552] 1. The server collects and analyzes the user's purchase history and transaction patterns.

[0553] 2. Based on the analysis results, the server suggests the most suitable payment method to the user.

[0554] 3. The user selects a suggested payment method and completes the payment on the terminal.

[0555] 4. When the payment deadline approaches, the server generates a reminder notification and sends it to the user's device.

[0556] Example of a prompt

[0557] Suggest the best payment method for user ID 12345, based on their first-quarter purchase history data.

[0558] This system frees users from the burden of creating documents and allows them to efficiently and securely select and manage payment methods.

[0559] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0560] Step 1:

[0561] The server collects user data (purchase history, transaction patterns, preferences, etc.). Specifically, it retrieves information such as the user's past email content, web browsing history, and purchase history from a database. This allows for the systematic accumulation of user data.

[0562] Step 2:

[0563] The server analyzes the collected user data. It uses natural language processing techniques to analyze the data and learn user behavior patterns and preferences. Specifically, it tokenizes text data and performs sentiment analysis and relationship extraction. It uses clustering techniques to classify the data into groups. This allows it to build a model for predicting future user behavior.

[0564] Step 3:

[0565] The server automatically performs tasks based on the learning results. Specifically, when a user gives an instruction such as "Create meeting materials for next week," the necessary information (date and time, format, and summary of content) is entered along with the instruction. The server refers to the learned model, automatically selects the optimal template, and generates the meeting materials using historical data.

[0566] Step 4:

[0567] The server sends the results of the executed task to the terminal. The generated meeting materials are sent from the server to the terminal and displayed to the user. This allows the user to review the generated materials.

[0568] Step 5:

[0569] Users review the generated documents and provide feedback. They input feedback, such as "I'd like this section described in more detail," through their device, and this feedback information is sent to the server.

[0570] Step 6:

[0571] The server analyzes the received feedback and corrects the task results. Specifically, based on the feedback, it uses natural language processing techniques to modify and add to the text, updating the document. A newly corrected document is then generated.

[0572] Step 7:

[0573] The revised document is sent again from the server to the terminal and displayed to the user. The user can then review the revised document.

[0574] Step 8:

[0575] The server analyzes the user's purchase history and transaction patterns to suggest the optimal payment method. Specifically, it uses clustering techniques and other machine learning algorithms to determine the best payment method (credit card, debit card, e-money, etc.) for the user.

[0576] Step 9:

[0577] The server sends payment methods to the user's terminal and displays them to the user. The user reviews the suggestions and selects a payment method.

[0578] Step 10:

[0579] The payment is processed on the terminal based on the payment method selected by the user. Specifically, the payment process is initiated and the payment is completed.

[0580] Step 11:

[0581] As the payment deadline approaches, the server automatically generates a reminder notification and sends it to the user's device. This helps users remember to make their payments.

[0582] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0583] System Overview

[0584] The present invention provides a system that efficiently performs user tasks using a user's personal agent, and in particular, offers more personalized assistance by combining it with an emotion engine that recognizes the user's emotions. The system includes processes for collecting, analyzing, learning from, automatically executing tasks, providing results, receiving feedback, and making corrections to user data.

[0585] Program Processing Overview

[0586] Introducing an emotional engine

[0587] The server collects data related to user behavior and preferences, as well as data to recognize the user's emotional state. Specifically, it analyzes emotions from the user's text, voice, and facial expressions, and uses an emotion engine to recognize this data as emotions. For example, it analyzes the user's facial expressions and tone of voice while they are reading an email to determine their stress level, joy, sadness, etc.

[0588] Learning Module

[0589] The server analyzes the collected user data and sentiment data. This allows it to learn user behavior patterns, preferences, and emotional states. Natural language processing techniques, clustering methods, and sentiment recognition algorithms are used in the analysis. Based on the obtained information, a model is built to address the user's emotional state. This model is stored on the server and made available to the agent in real time.

[0590] Task execution module

[0591] The terminal forwards user instructions to the server. If the user instructs, "Prepare meeting materials for next week," the server receives the instruction along with necessary information (date, time, format, summary of content) and the user's current emotional state. Based on learned behavioral patterns, preferences, and emotional states, the server selects the most suitable template and automatically executes the task. For example, if the user is stressed, the server may generate more concise and simpler materials. The generated results are then sent from the server back to the terminal.

[0592] Interface module

[0593] The device displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part to be described in more detail," that feedback is sent to the server via the device. The server analyzes the feedback and the user's emotional state and makes any necessary corrections. The corrected results are then sent back to the device and displayed to the user.

[0594] Explanation with specific examples

[0595] For preparing meeting materials for next week

[0596] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0597] 2. The device transmits instructions, necessary information, and user sentiment data to the server.

[0598] 3. The server references the user's past data and emotional state and selects an appropriate template.

[0599] 4. The server automatically generates meeting materials based on templates. For example, if a user is feeling stressed, it will generate a document that summarizes the information concisely.

[0600] 5. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0601] 6. Users review the materials and provide feedback as needed. For example, they might enter feedback such as, "I would like this section to be explained in more detail."

[0602] 7. The device transfers feedback and sentiment data to the server.

[0603] 8. The server analyzes feedback and sentiment data and modifies the document.

[0604] 9. The revised document is sent back to the terminal and displayed to the user.

[0605] This system not only frees users from the burden of document creation but also allows them to receive optimal support tailored to their emotional state. Agents automatically perform tasks and respond to the user's emotional state, significantly saving time and effort.

[0606] The following describes the processing flow.

[0607] Step 1:

[0608] The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0609] Step 2:

[0610] The device receives user instructions and simultaneously collects user emotional data (text, voice, facial expressions, etc.).

[0611] Step 3:

[0612] The device analyzes the user's emotional data and uses an emotion engine to recognize the user's current emotional state.

[0613] Step 4:

[0614] The terminal transmits the instructions, necessary information (date, time, format, and summary of the meeting), and emotional state to the server.

[0615] Step 5:

[0616] The server analyzes the instructions and emotional state received, and selects the optimal template based on the user's past data (past meeting materials, emails, preference data).

[0617] Step 6:

[0618] The server automatically fills in the necessary content based on the selected template and generates meeting materials. For example, if the user is under stress, it will create materials that are concise and easy to understand visually.

[0619] Step 7:

[0620] The server transfers the generated meeting materials to the terminal.

[0621] Step 8:

[0622] The terminal displays the meeting materials it has received to the user.

[0623] Step 9:

[0624] The user reviews the displayed meeting materials and enters any necessary feedback into the device, such as "Please elaborate on this section."

[0625] Step 10:

[0626] The device then collects user feedback and their emotional state at that time, and transmits it to the server.

[0627] Step 11:

[0628] The server analyzes the feedback and emotional state received and revises the meeting materials accordingly. For example, if the user is in an optimistic emotional state, it creates more comprehensive materials that include additional details.

[0629] Step 12:

[0630] The server resends the revised meeting materials to the terminal.

[0631] Step 13:

[0632] The terminal displays the revised meeting materials that have been resent to the user.

[0633] Step 14:

[0634] The user performs a final review and provides feedback again if necessary. This cycle continues until the document is finally finalized.

[0635] (Example 2)

[0636] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0637] Traditional personal agents could automate tasks based on user behavior patterns and preferences, but they could not consider the user's emotional state. As a result, they could not provide appropriate support depending on the user's emotional state, and in some cases, this could increase the user's stress. Furthermore, the automated execution of tasks and the incorporation of feedback often did not yield optimal results because they could not consider the emotional state. Therefore, there is a need for improved user satisfaction and efficient task execution.

[0638] In Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting user data, means for recognizing the user's emotional state using an emotion engine, means for analyzing the collected user data and emotional data to learn the user's behavioral patterns, preferences, and emotional state, means for automatically executing the user's tasks based on the learned behavioral patterns, preferences, and emotional state, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, and means for modifying the task results based on the received feedback and emotional data. This enables personalized support that corresponds to the user's emotional state, allowing for efficient task execution and improved user satisfaction.

[0639] "User data" refers to information such as user behavior, preferences, and instructions.

[0640] "Emotional data" refers to information about a user's emotional state extracted from their facial expressions, tone of voice, and text.

[0641] An "emotion engine" refers to a technology or software used to analyze and recognize a user's emotional state.

[0642] "Natural language processing technology" refers to the technology used to analyze text data and understand its meaning.

[0643] "Clustering techniques" refer to methods for grouping data according to similar characteristics.

[0644] A "generative AI model" refers to a model that uses artificial intelligence to generate text and other data.

[0645] A "machine learning algorithm" refers to an algorithm that learns from experience and recognizes patterns.

[0646] "User behavior patterns" refer to a user's tendencies to behave in specific situations.

[0647] "Preferences" refer to the things or tendencies that users like.

[0648] "Feedback" refers to opinions and requests for corrections provided by users.

[0649] This invention provides more personalized support in a system that efficiently performs user tasks using a user's personal agent, particularly by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[0650] Introducing an emotional engine

[0651] The server collects data related to user behavior and preferences, as well as data to recognize the user's emotional state. Specifically, it analyzes emotions from the user's text, voice, and facial expressions, and uses an emotion engine to recognize this data as emotions. For example, it analyzes the user's facial expressions and tone of voice while they are reading an email to determine their stress level, joy, sadness, etc.

[0652] Learning Module

[0653] The server analyzes the collected user data and sentiment data. This allows it to learn user behavior patterns, preferences, and emotional states. Natural language processing techniques, clustering methods, and sentiment recognition algorithms are used for the analysis. Based on the obtained information, a model is built to respond to the user's emotional state. This model is stored on the server and made available to the agent in real time.

[0654] Task execution module

[0655] The terminal forwards user instructions to the server. For example, if a user instructs the terminal to "create meeting materials for next week," the server receives the instruction along with necessary information (date and time, format, summary of content) and the user's current emotional state. The server selects the most suitable template based on learned behavioral patterns, preferences, and emotional states, and automatically executes the task. For example, if the user is stressed, the server may generate more concise and simpler materials. The generated results are then sent from the server back to the terminal.

[0656] Interface module

[0657] The device displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part described in more detail," that feedback is sent to the server via the device. The server analyzes the feedback and the user's emotional state and makes any necessary corrections. The corrected results are then sent back to the device and displayed to the user.

[0658] Specific example

[0659] Next, let's consider a specific example: preparing meeting materials for next week.

[0660] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0661] 2. The device transmits instructions, necessary information, and user sentiment data to the server.

[0662] 3. The server references the user's past data and emotional state and selects an appropriate template.

[0663] 4. The server automatically generates meeting materials based on templates. For example, if a user is feeling stressed, it will generate a document that summarizes the information concisely.

[0664] 5. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0665] 6. The user reviews the document and provides feedback as needed. For example, they might enter feedback such as, "I would like this section to be explained in more detail."

[0666] 7. The device transfers feedback and sentiment data to the server.

[0667] 8. The server revises the document based on the feedback and sends it back to the terminal. The final result is displayed to the user.

[0668] This system allows users to efficiently complete tasks and receive optimal support tailored to their emotions.

[0669] Example of a prompt

[0670] "Please prepare the meeting materials for next week. Please keep the information concise and to the best of your ability, taking your stress levels into consideration."

[0671] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0672] Step 1:

[0673] The user uses a device to input an instruction, such as "Prepare the meeting materials for next week." This input includes the instruction itself and necessary related information (e.g., date and time, format, and summary of the content). The device recognizes this input and also determines the user's current emotional state (collected from facial expressions, voice, etc.). The input data includes the instruction text and emotional data. This data is temporarily stored within the device.

[0674] Step 2:

[0675] The terminal transfers collected instruction content and sentiment data to the server. Input includes instruction text, date and time, format, content summary, and sentiment data. Output after transfer is achieved when the server receives this data. Specifically, the terminal sends data to the server using network communication.

[0676] Step 3:

[0677] The server analyzes the received instruction text and sentiment data. Natural language processing techniques are used for the analysis. The input data consists of instruction text and sentiment data. The analysis results in an understanding of the instruction content and a determination of the emotional state. The output includes the analysis results and the user's emotional state. Specifically, the server executes a text analysis engine and an sentiment recognition algorithm.

[0678] Step 4:

[0679] The server selects an appropriate template based on a model that has learned past behavioral patterns, preferences, and emotional states. Input data includes analysis results and historical data. This data is evaluated using clustering techniques to determine the optimal template. The output is the selected template. Specifically, the server retrieves historical data from the database, applies the machine learning model, and selects a template.

[0680] Step 5:

[0681] The server creates documents using the selected template. A generative AI model is used here. The input data includes the template and the necessary information. Since the generated documents also take emotional states into account, for example, a concise document will be generated for a user experiencing stress. The output is the generated document. Specifically, the server runs the generative AI model and embeds information into the template.

[0682] Step 6:

[0683] The server sends the generated document to the terminal. The data sent is the generated document. Specifically, the server sends the data to the terminal using network communication. The output is the terminal receiving the document.

[0684] Step 7:

[0685] The terminal displays the received materials to the user. The input data is generated material sent from the server. Specifically, the terminal uses its screen display function to visually provide the materials to the user. The output is that the materials become viewable by the user.

[0686] Step 8:

[0687] The user reviews the document and enters feedback. The input data consists of user correction instructions and additional information. Specifically, the user enters text using a terminal. The output is the feedback text.

[0688] Step 9:

[0689] The terminal retransmits the input feedback and sentiment data to the server. The input data consists of feedback text and sentiment data. Specifically, the terminal sends data to the server using network communication. The output is the server receiving this data again.

[0690] Step 10:

[0691] The server analyzes user feedback and sentiment data to revise the document. Input data consists of feedback text and sentiment data. The analysis determines the necessary revisions to the document. The output is the revised document. Specifically, the server runs the AI ​​model again to update the document.

[0692] Step 11:

[0693] The server resends the corrected document to the terminal. The input data is the corrected document. Specifically, the server sends the data to the terminal using network communication. The output is the terminal receiving the corrected document.

[0694] Step 12:

[0695] The terminal redisplays the revised document to the user. The input data is the revised document. Specifically, the terminal uses its screen display function to visually present the document to the user again. The output is that the revised document is made viewable by the user.

[0696] In this way, the system can efficiently perform tasks while taking the user's emotional state into consideration.

[0697] (Application Example 2)

[0698] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0699] Traditional personal agent systems provided the functionality to automate tasks based on user behavior patterns and preferences, but they did not take user emotions into consideration. As a result, users may experience stress or dissatisfaction. Especially in physical stores, responding to customers' emotional states is crucial, but this was difficult with current systems. Therefore, there was a need for technology that could provide more appropriate and personalized customer service based on customer emotions.

[0700] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0701] In this invention, the server includes means for collecting user data, means for analyzing the collected user data and emotional data to learn the user's behavioral patterns, preferences, and emotional states, and means for automatically executing user tasks based on the learned behavioral patterns, preferences, and emotional states. This makes it possible to detect the customer's emotional state in real time and provide optimal advice accordingly.

[0702] "User data" refers to all information about a user, including their behavior, preferences, instructions, and feedback.

[0703] "Emotional data" refers to information about a user's emotional state, analyzed from their facial expressions, voice, text, etc.

[0704] "Analysis" refers to the process of evaluating and classifying collected data to extract meaningful information.

[0705] "Behavioral patterns" refer to the tendencies of actions and habits that users repeatedly perform.

[0706] "Preferences" refer to a user's likes and dislikes, or their tendencies to prefer certain things.

[0707] "Learning" refers to the process where an algorithm uses data to build a model of user behavior patterns and preferences, and then uses that model for prediction and decision-making.

[0708] A "task" refers to a series of actions or procedures performed based on instructions from a user.

[0709] "Execute automatically" means that the system performs a pre-configured process without user intervention.

[0710] "Terminal" refers to information processing devices such as computers, smartphones, and tablets used by users.

[0711] "Feedback" refers to evaluations and opinions provided by users regarding the results.

[0712] "Correction" refers to making improvements or changes to the initial results based on the feedback provided.

[0713] "Detecting customer emotional states in real time" means recognizing customer emotions instantly, without any time lag.

[0714] "Advice" refers to the recommended actions or instructions provided by the system.

[0715] This invention realizes a system that combines an emotion engine with a user's personal agent to analyze the emotional state of the user and customer and provide optimal advice accordingly. Each part of the system operates in response to the server, terminal, and user.

[0716] System Overview

[0717] The server has the capability to collect and analyze user data and sentiment data. This utilizes natural language processing techniques and sentiment recognition algorithms. Specifically, software such as TensorFlow, OpenCV, and SpeechRecognition are used. The collected data is used to learn the user's behavioral patterns, preferences, and emotional states using machine learning algorithms. The server also automatically performs user tasks based on the learned model and sends the results to the user's device.

[0718] The device displays results sent from the server to the user and receives real-time feedback. It also plays a role in transferring the user's emotional data to the server. Smart glasses are equipped with a camera and microphone, which can capture the user's (or customer's) facial expressions and voice. This allows for real-time analysis of customer emotions and provides staff with the most appropriate advice.

[0719] Program Processing Overview

[0720] The server collects and analyzes user data and identifies emotional data using an emotion engine. For example, it can analyze emotions from a user's text, voice, and facial expressions, and recognize their emotional state in real time. The server also uses machine learning algorithms to learn user behavior patterns and preferences from the collected data. Based on this, it automatically executes tasks and sends the results to the terminal.

[0721] The terminal displays the results sent from the server to the user and receives user feedback. The feedback is sent to the server for further analysis and correction. The corrected results are then sent back to the terminal and displayed to the user.

[0722] As a concrete example, consider the following scenario: While a customer is examining a product, a staff member wears smart glasses. The system analyzes the customer's facial expression, and if it detects a satisfied expression, the staff member's glasses display the advice, "The customer is happy. Let's suggest additional products." In another scenario, if the customer appears confused, the system provides the advice, "The customer is confused. Let's provide a more detailed explanation."

[0723] An example of a specific prompt for a generative AI model is: "Please provide a prompt for a generative AI model to recognize a customer's smile and advise on how to respond if the customer is happy." This will enable the AI ​​model to automatically generate appropriate advice.

[0724] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0725] Step 1:

[0726] The server collects user and sentiment data. Specifically, it collects data such as user and customer behavior, preferences, facial expressions, voice, and text. This input data is sent to the server and stored in a database. This data collection provides initial information for use throughout the system.

[0727] Step 2:

[0728] The server analyzes the collected user data and emotion data. Natural language processing techniques and emotion recognition algorithms are used in this process. Specifically, machine learning libraries such as Python's TensorFlow are used to analyze text data, and OpenCV is used to analyze facial expression data. The analysis results are extracted as user behavior patterns, preferences, and emotional states.

[0729] Step 3:

[0730] Based on the analysis results, the server uses machine learning algorithms to model and learn user behavior patterns, preferences, and emotional states. This generates a predictive model for future task performance. The model's accuracy improves through sequential feedback during this learning process.

[0731] Step 4:

[0732] The user or device instructs the server to perform a specific task. For example, it might instruct the server to "prepare the meeting materials for next week." This instruction includes information such as the date and time, format, and a summary of the content, as well as the user's current emotional state. This becomes the input sent to the server.

[0733] Step 5:

[0734] The server automatically performs tasks requested by the user based on a trained model. For example, if the user is stressed, it automatically generates simple and easy-to-understand meeting materials. In this process, it uses a generative AI model to select and process appropriate templates and content, and then outputs the final materials.

[0735] Step 6:

[0736] The server sends the task results to the terminal. The generated materials and advice are delivered to the user's terminal and displayed to the user. At this point, output to the user is complete, and the user can review the results as needed.

[0737] Step 7:

[0738] Users provide feedback on the results displayed on their devices. For example, they can send detailed requests to the server via their devices, such as "Please describe this part in more detail." This becomes the user's input data.

[0739] Step 8:

[0740] The server analyzes the received feedback and makes necessary corrections. Specifically, it uses machine learning algorithms again to adjust the task results, taking the feedback into consideration, and makes appropriate corrections. The corrected results are then sent back to the terminal.

[0741] Step 9:

[0742] The device then redisplays the corrected results to the user. This is where final confirmation takes place, and the user can provide further feedback on the results. This ensures that the optimal result, tailored to the user's requirements down to the smallest detail, is achieved.

[0743] Specific actions

[0744] The smart glasses capture the customer's facial expressions and voice in real time using their camera and microphone.

[0745] The acquired data is sent to a server and analyzed using natural language processing technology and emotion recognition algorithms.

[0746] If the customer is happy, advise the staff, "The customer is happy. Let's suggest additional products."

[0747] If the customer is confused, advise them to "The customer is confused. Let's provide a detailed explanation."

[0748] An example of a specific prompt for a generative AI model is: "Please provide a prompt for a generative AI model to recognize a customer's smile and advise on how to respond if the customer is happy." This will enable the AI ​​model to automatically generate appropriate advice.

[0749] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0750] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0751] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0752] [Third Embodiment]

[0753] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0754] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0755] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0756] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0757] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0758] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0759] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0760] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0761] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0762] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0763] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0764] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0765] System Overview

[0766] The present invention is a system that efficiently performs user tasks using the user's personal agent, and specifically includes processes for collecting, analyzing, learning, automatically executing tasks, providing results, receiving feedback, and making corrections based on user data.

[0767] Program Processing Overview

[0768] Learning Module

[0769] The server first collects user data related to the user's behavior and preferences. This includes, for example, the user's past email content, web browsing history, and purchase history. Then, this data is analyzed using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences. This builds a model for predicting the user's future behavior. The model is stored on the server and made available to the agent in real time.

[0770] Task execution module

[0771] The terminal forwards user instructions to the server. For example, if a user instructs the server to "create meeting materials for next week," the server receives the instruction along with the necessary information (date and time, format, and summary of content). The server selects the optimal template based on learned behavioral patterns and preferences and automatically executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are then sent from the server to the terminal.

[0772] Interface module

[0773] The terminal displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part to be described in more detail," that feedback is sent to the server via the terminal. The server analyzes the feedback and makes any necessary corrections. The corrected results are then sent back to the terminal and displayed to the user.

[0774] Explanation with specific examples

[0775] For preparing meeting materials for next week

[0776] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0777] 2. The terminal forwards the instructions to the server, which then selects an appropriate template by referring to past meeting materials and user preferences.

[0778] 3. The server automatically generates meeting materials based on the template.

[0779] 4. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0780] 5. The user reviews the document and enters feedback into the device, such as "I would like this part added."

[0781] 6. The terminal forwards the feedback to the server, which analyzes the feedback and corrects the document.

[0782] 7. The revised document is sent back to the terminal and displayed to the user.

[0783] This frees users from the burden of document creation, allowing them to efficiently focus on important tasks. By having agents automatically perform tasks, users can save significant time and effort.

[0784] The following describes the processing flow.

[0785] Step 1:

[0786] The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0787] Step 2:

[0788] The terminal receives user instructions and transmits those instructions along with necessary information (such as the date, time, format, and summary of the meeting) to the server.

[0789] Step 3:

[0790] The server analyzes the received instructions and selects the most suitable template based on the user's past data (past meeting materials, emails, preference data).

[0791] Step 4:

[0792] The server automatically generates meeting materials by filling in the necessary content based on the selected template.

[0793] Step 5:

[0794] The server transfers the generated meeting materials to the terminal.

[0795] Step 6:

[0796] The terminal displays the meeting materials it has received to the user.

[0797] Step 7:

[0798] The user reviews the displayed meeting materials and enters any necessary feedback into the device, such as "Please elaborate on this section."

[0799] Step 8:

[0800] The device receives user feedback and forwards it to the server.

[0801] Step 9:

[0802] The server analyzes the feedback received and makes revisions to the meeting materials.

[0803] Step 10:

[0804] The server resends the revised meeting materials to the terminal.

[0805] Step 11:

[0806] The terminal displays the revised meeting materials that have been resent to the user.

[0807] Step 12:

[0808] The user performs a final review and provides feedback again if necessary. This cycle continues until the document is finally finalized.

[0809] (Example 1)

[0810] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0811] Traditional task management systems were inefficient because users had to manually set tasks, check the results, and make corrections if necessary. Furthermore, it was difficult to optimize task execution based on individual user preferences and behavioral patterns. Therefore, there was a need for a system that would reduce the burden on users and allow for more efficient task execution.

[0812] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0813] In this invention, the server includes means for collecting user data, means for analyzing the collected user data using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences, means for automatically executing the user's tasks based on the learned behavior patterns and preferences, selecting the optimal template and generating results, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, and means for analyzing the received feedback and modifying the task results. This enables the automation of tasks based on the user's individual preferences and behavior patterns, and adaptive modification based on feedback.

[0814] "User data" refers to information related to a user's behavior and preferences, such as the content of emails, web browsing history, and purchase history.

[0815] "Natural language processing technology" refers to the technology that uses computers to understand, generate, and analyze human language, such as text and audio data.

[0816] "Clustering techniques" refer to analytical methods that group similar data together to understand the structure of the data.

[0817] "Behavioral patterns" refer to the tendencies of a series of actions and operations that a user has performed in the past.

[0818] "Preferences" refer to the individual user's tastes and preferences.

[0819] A "template" refers to a pre-prepared format or template for performing a specific task.

[0820] "Feedback" refers to comments and suggestions for improvements provided by users regarding the results.

[0821] A "machine learning algorithm" refers to a mathematical method used to learn patterns based on data and perform tasks such as prediction and classification.

[0822] Modes for carrying out the invention

[0823] This invention provides a system for collecting, analyzing, learning from, and automatically executing user data, thereby reducing the burden on users and efficiently carrying out tasks. The specific configuration and operation of this system are described below.

[0824] System Configuration

[0825] This system consists of a server for collecting and analyzing user data, and terminals where users input instructions. The server is a high-performance computer system that uses databases, natural language processing techniques (e.g., BERT and GPT-3), clustering methods (e.g., K-means), and machine learning algorithms. The terminals are devices that users can directly operate (e.g., PCs, smartphones, tablets).

[0826] Program Processing Overview

[0827] The server first collects data related to user behavior and preferences. Specifically, it obtains data through email servers, browser logs, and APIs of online shopping sites. This data includes past email content, web browsing history, and purchase history.

[0828] Next, the server analyzes the collected data using natural language processing techniques (e.g., BERT, GPT-3) and clustering methods (e.g., K-means). Through this analysis, user behavior patterns and preferences are learned, and individual user models are constructed.

[0829] When a user enters a task into the terminal, the terminal forwards this instruction to the server. For example, if a user gives the instruction "Prepare the meeting materials for next week," the server will send the instruction along with the necessary information (date and time, format, and summary of content).

[0830] The server selects the optimal template based on a trained user model and executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are sent from the server to the terminal, which then displays the materials to the user.

[0831] When a user reviews a document and provides feedback as needed, the device forwards this feedback to the server. The server analyzes the feedback and makes any necessary corrections. The corrected document is then sent back to the device and displayed to the user.

[0832] Specific example

[0833] For example, in a task to create meeting materials for the following week, the user inputs the instruction "Create meeting materials for next week" using their device. The device forwards this instruction to the server, which selects an appropriate template by referring to past meeting materials and the user's preferences. Based on this template, the server automatically generates the meeting materials and sends them to the device. When the user reviews the materials and provides feedback such as "I would like this section to be explained in more detail," the server analyzes the feedback and revises the materials. The revised materials are then sent to the device and displayed to the user.

[0834] Example of a prompt

[0835] An example of a prompt message is, "GPT-3, if the user asks you to create meeting materials for next week, generate new materials using past materials as a reference." By using prompts like this, the generative AI model can appropriately perform the task requested by the user.

[0836] As described above, the system of the present invention can automatically perform tasks based on the user's preferences and behavioral patterns, and can make adaptive modifications based on feedback.

[0837] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0838] Step 1: Collecting User Data

[0839] The server collects data related to the user's past behavior and preferences.

[0840] Specific operation: The server connects to the mail server API and retrieves email data from the past six months. It also collects browser browsing history data from log files and retrieves purchase history from shopping sites via API.

[0841] Input: Data from mail servers, browser log files, and shopping site APIs.

[0842] Output: The user's email content, web browsing history, and purchase history are stored in the server's database.

[0843] Step 2: Analyze and learn from user data

[0844] The server analyzes the collected data and learns the user's behavior patterns and preferences.

[0845] Specific operation: The server uses a natural language processing library (e.g., BERT, GPT-3) to extract important keywords from email content. Next, it applies a clustering algorithm (e.g., K-means) to classify web browsing history into different categories.

[0846] Input: User data collected in Step 1.

[0847] Output: The analyzed behavioral patterns and preferences are saved as a user model.

[0848] Step 3: Receiving the Task

[0849] The user enters the task details via their terminal. The task is then transferred to the server.

[0850] Specific action: The user enters "Create meeting materials for next week" into the application's form and presses the submit button. The device sends this data to the server as an HTTP request.

[0851] Input: Task instructions entered by the user on the device.

[0852] Output: The task instruction reaches the server and is added to the processing queue.

[0853] Step 4: Task execution and result generation

[0854] The server executes tasks and generates results based on the user's behavior patterns and preferences.

[0855] Specific operation: The server selects the most suitable meeting material template from user preference data and automatically generates new materials based on past meeting material data. It adds necessary context and content using a generation AI model (e.g., GPT-3).

[0856] Input: User model, past meeting materials, templates.

[0857] Output: The generated meeting materials file is saved to the server.

[0858] Step 5: Providing Results

[0859] The terminal displays the results received from the server to the user.

[0860] Specific operation: The server sends the generated meeting materials file to the terminal, and the terminal's application displays this file in the user interface.

[0861] Input: Meeting materials file sent from the server.

[0862] Output: The user is able to view the meeting materials on their device.

[0863] Step 6: Receiving Feedback

[0864] Users review the materials and provide feedback.

[0865] Specific action: The user enters feedback into a form on their device, stating "I would like this part described in more detail," and presses the submit button. The device then sends this feedback to the server.

[0866] Input: Feedback entered by the user on the device.

[0867] Output: Feedback reaches the server and is analyzed.

[0868] Step 7: Corrections based on feedback

[0869] The server analyzes the feedback it receives and corrects the document.

[0870] Specific operation: The server analyzes the feedback using a natural language processing tool and makes necessary corrections using a generation AI model. The corrected document is then regenerated and sent to the terminal.

[0871] Input: User feedback.

[0872] Output: The revised meeting materials are sent to the terminal and displayed to the user again.

[0873] The above outlines the specific processing steps of the system. This series of steps enables automated task execution and adaptive correction, significantly reducing the burden on the user.

[0874] (Application Example 1)

[0875] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0876] In modern society, the series of tasks and payment management that users perform daily have become increasingly complex, and optimization is needed. In particular, there is a lack of efficient and secure payment methods that suggest the optimal payment method based on the user's purchase history and transaction patterns, which often leads to users wasting time and effort. To address this challenge, a system is needed that collects and analyzes user data and uses trained models to efficiently automate tasks.

[0877] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0878] In this invention, the server includes means for collecting user data, means for analyzing the collected user data and learning the user's behavior patterns and preferences, means for automatically executing the user's tasks based on the learned behavior patterns and preferences, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, means for modifying the task results based on the received feedback, means for analyzing the user's purchase history and transaction patterns and suggesting the optimal payment method, means for generating reminder notifications based on the payment method, and means for quickly processing payments on the user's terminal. This enables efficient and secure task management and payment processing for users, saving them time and effort.

[0879] "User data" refers to various types of information related to a user, such as their purchase history, transaction patterns, and preferences.

[0880] "Analysis" refers to the process of analyzing collected user data to reveal user behavior patterns and preferences.

[0881] "Behavioral patterns" refer to the tendencies and habits of how a user behaves in specific situations.

[0882] "Preferences" refer to the user's tendency to choose things and services that they particularly like.

[0883] A "task" refers to a specific task or operation that a user is required to perform.

[0884] "Device" refers to an electronic device used by a user, and includes smartphones, computers, and other similar devices.

[0885] "Feedback" refers to opinions, evaluations, and requests for corrections provided by users.

[0886] "Modification" refers to the act of changing task results, settings, etc., based on user feedback.

[0887] "Purchase history" refers to information about the products and services that a user has purchased in the past.

[0888] "Trading patterns" refer to the tendencies and habits of a user in their trading.

[0889] "Payment method" refers to the payment method selected by the user, and includes credit cards, debit cards, electronic money, etc.

[0890] A "reminder notification" refers to an automated notification that informs the user of specific tasks, payment deadlines, or other important information.

[0891] "Payment processing" refers to the act of actually completing a monetary transaction using the selected payment method.

[0892] Modes for carrying out the invention

[0893] System Overview

[0894] This invention is a system that efficiently performs user tasks using a user's personal agent. Specifically, it includes processes for collecting, analyzing, and learning user data, automatically executing tasks, providing results, receiving feedback, and making corrections. It also includes analyzing the user's purchase history and transaction patterns, suggesting the optimal payment method, generating payment reminder notifications, and processing them quickly on the terminal.

[0895] Program Processing Overview

[0896] Learning Module

[0897] The server first collects user data related to the user's behavior and preferences. This includes the user's past email content, web browsing history, and purchase history. Next, the collected data is analyzed using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences. Using these learning results, a model is built to predict the user's future behavior. The model is stored on the server and made available to the agent in real time.

[0898] Task execution module

[0899] The terminal forwards user instructions to the server. For example, if a user instructs the server to "create meeting materials for next week," the server receives the instruction along with the necessary information (date and time, format, and summary of content). The server selects the optimal template based on learned behavioral patterns and preferences and automatically executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are then sent from the server to the terminal.

[0900] Interface module

[0901] The terminal displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user enters feedback such as "I would like this part described in more detail" into the terminal, that feedback is sent to the server. The server analyzes the feedback, makes any necessary corrections, and sends the corrected results back to the terminal.

[0902] Payment management module

[0903] The server collects and analyzes the user's purchase history and transaction patterns to suggest the most suitable payment method. For example, it suggests the best payment method from options such as credit cards, debit cards, and electronic money based on the user's past usage. Furthermore, if a payment deadline is approaching, it generates a reminder notification and automatically sends it to the user's device.

[0904] Specific example

[0905] For preparing meeting materials for next week

[0906] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0907] 2. The terminal forwards the instructions to the server, which then selects an appropriate template by referring to past meeting materials and user preferences.

[0908] 3. The server automatically generates meeting materials based on the template.

[0909] 4. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0910] 5. The user reviews the document and enters feedback into the device, such as "I would like this part added."

[0911] 6. The terminal forwards the feedback to the server, which analyzes the feedback and corrects the document.

[0912] 7. The revised document is sent back to the terminal and displayed to the user.

[0913] In the case of payment management

[0914] 1. The server collects and analyzes the user's purchase history and transaction patterns.

[0915] 2. Based on the analysis results, the server suggests the most suitable payment method to the user.

[0916] 3. The user selects a suggested payment method and completes the payment on the terminal.

[0917] 4. When the payment deadline approaches, the server generates a reminder notification and sends it to the user's device.

[0918] Example of a prompt

[0919] Suggest the best payment method for user ID 12345, based on their first-quarter purchase history data.

[0920] This system frees users from the burden of creating documents and allows them to efficiently and securely select and manage payment methods.

[0921] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0922] Step 1:

[0923] The server collects user data (purchase history, transaction patterns, preferences, etc.). Specifically, it retrieves information such as the user's past email content, web browsing history, and purchase history from a database. This allows for the systematic accumulation of user data.

[0924] Step 2:

[0925] The server analyzes the collected user data. It uses natural language processing techniques to analyze the data and learn user behavior patterns and preferences. Specifically, it tokenizes text data and performs sentiment analysis and relationship extraction. It uses clustering techniques to classify the data into groups. This allows it to build a model for predicting future user behavior.

[0926] Step 3:

[0927] The server automatically performs tasks based on the learning results. Specifically, when a user gives an instruction such as "Create meeting materials for next week," the necessary information (date and time, format, and summary of content) is entered along with the instruction. The server refers to the learned model, automatically selects the optimal template, and generates the meeting materials using historical data.

[0928] Step 4:

[0929] The server sends the results of the executed task to the terminal. The generated meeting materials are sent from the server to the terminal and displayed to the user. This allows the user to review the generated materials.

[0930] Step 5:

[0931] Users review the generated documents and provide feedback. They input feedback, such as "I'd like this section described in more detail," through their device, and this feedback information is sent to the server.

[0932] Step 6:

[0933] The server analyzes the received feedback and corrects the task results. Specifically, based on the feedback, it uses natural language processing techniques to modify and add to the text, updating the document. A newly corrected document is then generated.

[0934] Step 7:

[0935] The revised document is sent again from the server to the terminal and displayed to the user. The user can then review the revised document.

[0936] Step 8:

[0937] The server analyzes the user's purchase history and transaction patterns to suggest the optimal payment method. Specifically, it uses clustering techniques and other machine learning algorithms to determine the best payment method (credit card, debit card, e-money, etc.) for the user.

[0938] Step 9:

[0939] The server sends payment methods to the user's terminal and displays them to the user. The user reviews the suggestions and selects a payment method.

[0940] Step 10:

[0941] The payment is processed on the terminal based on the payment method selected by the user. Specifically, the payment process is initiated and the payment is completed.

[0942] Step 11:

[0943] As the payment deadline approaches, the server automatically generates a reminder notification and sends it to the user's device. This helps users remember to make their payments.

[0944] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0945] System Overview

[0946] The present invention provides a system that efficiently performs user tasks using a user's personal agent, and in particular, offers more personalized assistance by combining it with an emotion engine that recognizes the user's emotions. The system includes processes for collecting, analyzing, learning from, automatically executing tasks, providing results, receiving feedback, and making corrections to user data.

[0947] Program Processing Overview

[0948] Introducing an emotional engine

[0949] The server collects data related to user behavior and preferences, as well as data to recognize the user's emotional state. Specifically, it analyzes emotions from the user's text, voice, and facial expressions, and uses an emotion engine to recognize this data as emotions. For example, it analyzes the user's facial expressions and tone of voice while they are reading an email to determine their stress level, joy, sadness, etc.

[0950] Learning Module

[0951] The server analyzes the collected user data and sentiment data. This allows it to learn user behavior patterns, preferences, and emotional states. Natural language processing techniques, clustering methods, and sentiment recognition algorithms are used in the analysis. Based on the obtained information, a model is built to address the user's emotional state. This model is stored on the server and made available to the agent in real time.

[0952] Task execution module

[0953] The terminal forwards user instructions to the server. If the user instructs, "Prepare meeting materials for next week," the server receives the instruction along with necessary information (date, time, format, summary of content) and the user's current emotional state. Based on learned behavioral patterns, preferences, and emotional states, the server selects the most suitable template and automatically executes the task. For example, if the user is stressed, the server may generate more concise and simpler materials. The generated results are then sent from the server back to the terminal.

[0954] Interface module

[0955] The device displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part to be described in more detail," that feedback is sent to the server via the device. The server analyzes the feedback and the user's emotional state and makes any necessary corrections. The corrected results are then sent back to the device and displayed to the user.

[0956] Explanation with specific examples

[0957] For preparing meeting materials for next week

[0958] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0959] 2. The device transmits instructions, necessary information, and user sentiment data to the server.

[0960] 3. The server references the user's past data and emotional state and selects an appropriate template.

[0961] 4. The server automatically generates meeting materials based on templates. For example, if a user is feeling stressed, it will generate a document that summarizes the information concisely.

[0962] 5. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[0963] 6. Users review the materials and provide feedback as needed. For example, they might enter feedback such as, "I would like this section to be explained in more detail."

[0964] 7. The device transfers feedback and sentiment data to the server.

[0965] 8. The server analyzes feedback and sentiment data and modifies the document.

[0966] 9. The revised document is sent back to the terminal and displayed to the user.

[0967] This system not only frees users from the burden of document creation but also allows them to receive optimal support tailored to their emotional state. Agents automatically perform tasks and respond to the user's emotional state, significantly saving time and effort.

[0968] The following describes the processing flow.

[0969] Step 1:

[0970] The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[0971] Step 2:

[0972] The device receives user instructions and simultaneously collects user emotional data (text, voice, facial expressions, etc.).

[0973] Step 3:

[0974] The device analyzes the user's emotional data and uses an emotion engine to recognize the user's current emotional state.

[0975] Step 4:

[0976] The terminal transmits the instructions, necessary information (date, time, format, and summary of the meeting), and emotional state to the server.

[0977] Step 5:

[0978] The server analyzes the instructions and emotional state received, and selects the optimal template based on the user's past data (past meeting materials, emails, preference data).

[0979] Step 6:

[0980] The server automatically fills in the necessary content based on the selected template and generates meeting materials. For example, if the user is under stress, it will create materials that are concise and easy to understand visually.

[0981] Step 7:

[0982] The server transfers the generated meeting materials to the terminal.

[0983] Step 8:

[0984] The terminal displays the meeting materials it has received to the user.

[0985] Step 9:

[0986] The user reviews the displayed meeting materials and enters any necessary feedback into the device, such as "Please elaborate on this section."

[0987] Step 10:

[0988] The device then collects user feedback and their emotional state at that time, and transmits it to the server.

[0989] Step 11:

[0990] The server analyzes the feedback and emotional state received and revises the meeting materials accordingly. For example, if the user is in an optimistic emotional state, it creates more comprehensive materials that include additional details.

[0991] Step 12:

[0992] The server resends the revised meeting materials to the terminal.

[0993] Step 13:

[0994] The terminal displays the revised meeting materials that have been resent to the user.

[0995] Step 14:

[0996] The user performs a final review and provides feedback again if necessary. This cycle continues until the document is finally finalized.

[0997] (Example 2)

[0998] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0999] Traditional personal agents could automate tasks based on user behavior patterns and preferences, but they could not consider the user's emotional state. As a result, they could not provide appropriate support depending on the user's emotional state, and in some cases, this could increase the user's stress. Furthermore, the automated execution of tasks and the incorporation of feedback often did not yield optimal results because they could not consider the emotional state. Therefore, there is a need for improved user satisfaction and efficient task execution.

[1000] In Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting user data, means for recognizing the user's emotional state using an emotion engine, means for analyzing the collected user data and emotional data to learn the user's behavioral patterns, preferences, and emotional state, means for automatically executing the user's tasks based on the learned behavioral patterns, preferences, and emotional state, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, and means for modifying the task results based on the received feedback and emotional data. This enables personalized support that corresponds to the user's emotional state, allowing for efficient task execution and improved user satisfaction.

[1001] "User data" refers to information such as user behavior, preferences, and instructions.

[1002] "Emotional data" refers to information about a user's emotional state extracted from their facial expressions, tone of voice, and text.

[1003] An "emotion engine" refers to a technology or software used to analyze and recognize a user's emotional state.

[1004] "Natural language processing technology" refers to the technology used to analyze text data and understand its meaning.

[1005] "Clustering techniques" refer to methods for grouping data according to similar characteristics.

[1006] A "generative AI model" refers to a model that uses artificial intelligence to generate text and other data.

[1007] A "machine learning algorithm" refers to an algorithm that learns from experience and recognizes patterns.

[1008] "User behavior patterns" refer to a user's tendencies to behave in specific situations.

[1009] "Preferences" refer to the things or tendencies that users like.

[1010] "Feedback" refers to opinions and requests for corrections provided by users.

[1011] This invention provides more personalized support in a system that efficiently performs user tasks using a user's personal agent, particularly by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[1012] Introducing an emotional engine

[1013] The server collects data related to user behavior and preferences, as well as data to recognize the user's emotional state. Specifically, it analyzes emotions from the user's text, voice, and facial expressions, and uses an emotion engine to recognize this data as emotions. For example, it analyzes the user's facial expressions and tone of voice while they are reading an email to determine their stress level, joy, sadness, etc.

[1014] Learning Module

[1015] The server analyzes the collected user data and sentiment data. This allows it to learn user behavior patterns, preferences, and emotional states. Natural language processing techniques, clustering methods, and sentiment recognition algorithms are used for the analysis. Based on the obtained information, a model is built to respond to the user's emotional state. This model is stored on the server and made available to the agent in real time.

[1016] Task execution module

[1017] The terminal forwards user instructions to the server. For example, if a user instructs the terminal to "create meeting materials for next week," the server receives the instruction along with necessary information (date and time, format, summary of content) and the user's current emotional state. The server selects the most suitable template based on learned behavioral patterns, preferences, and emotional states, and automatically executes the task. For example, if the user is stressed, the server may generate more concise and simpler materials. The generated results are then sent from the server back to the terminal.

[1018] Interface module

[1019] The device displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part described in more detail," that feedback is sent to the server via the device. The server analyzes the feedback and the user's emotional state and makes any necessary corrections. The corrected results are then sent back to the device and displayed to the user.

[1020] Specific example

[1021] Next, let's consider a specific example: preparing meeting materials for next week.

[1022] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[1023] 2. The device transmits instructions, necessary information, and user sentiment data to the server.

[1024] 3. The server references the user's past data and emotional state and selects an appropriate template.

[1025] 4. The server automatically generates meeting materials based on templates. For example, if a user is feeling stressed, it will generate a document that summarizes the information concisely.

[1026] 5. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[1027] 6. The user reviews the document and provides feedback as needed. For example, they might enter feedback such as, "I would like this section to be explained in more detail."

[1028] 7. The device transfers feedback and sentiment data to the server.

[1029] 8. The server revises the document based on the feedback and sends it back to the terminal. The final result is displayed to the user.

[1030] This system allows users to efficiently complete tasks and receive optimal support tailored to their emotions.

[1031] Example of a prompt

[1032] "Please prepare the meeting materials for next week. Please keep the information concise and to the best of your ability, taking your stress levels into consideration."

[1033] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1034] Step 1:

[1035] The user uses a device to input an instruction, such as "Prepare the meeting materials for next week." This input includes the instruction itself and necessary related information (e.g., date and time, format, and summary of the content). The device recognizes this input and also determines the user's current emotional state (collected from facial expressions, voice, etc.). The input data includes the instruction text and emotional data. This data is temporarily stored within the device.

[1036] Step 2:

[1037] The terminal transfers collected instruction content and sentiment data to the server. Input includes instruction text, date and time, format, content summary, and sentiment data. Output after transfer is achieved when the server receives this data. Specifically, the terminal sends data to the server using network communication.

[1038] Step 3:

[1039] The server analyzes the received instruction text and sentiment data. Natural language processing techniques are used for the analysis. The input data consists of instruction text and sentiment data. The analysis results in an understanding of the instruction content and a determination of the emotional state. The output includes the analysis results and the user's emotional state. Specifically, the server executes a text analysis engine and an sentiment recognition algorithm.

[1040] Step 4:

[1041] The server selects an appropriate template based on a model that has learned past behavioral patterns, preferences, and emotional states. Input data includes analysis results and historical data. This data is evaluated using clustering techniques to determine the optimal template. The output is the selected template. Specifically, the server retrieves historical data from the database, applies the machine learning model, and selects a template.

[1042] Step 5:

[1043] The server creates documents using the selected template. A generative AI model is used here. The input data includes the template and the necessary information. Since the generated documents also take emotional states into account, for example, a concise document will be generated for a user experiencing stress. The output is the generated document. Specifically, the server runs the generative AI model and embeds information into the template.

[1044] Step 6:

[1045] The server sends the generated document to the terminal. The data sent is the generated document. Specifically, the server sends the data to the terminal using network communication. The output is the terminal receiving the document.

[1046] Step 7:

[1047] The terminal displays the received materials to the user. The input data is generated material sent from the server. Specifically, the terminal uses its screen display function to visually provide the materials to the user. The output is that the materials become viewable by the user.

[1048] Step 8:

[1049] The user reviews the document and enters feedback. The input data consists of user correction instructions and additional information. Specifically, the user enters text using a terminal. The output is the feedback text.

[1050] Step 9:

[1051] The terminal retransmits the input feedback and sentiment data to the server. The input data consists of feedback text and sentiment data. Specifically, the terminal sends data to the server using network communication. The output is the server receiving this data again.

[1052] Step 10:

[1053] The server analyzes user feedback and sentiment data to revise the document. Input data consists of feedback text and sentiment data. The analysis determines the necessary revisions to the document. The output is the revised document. Specifically, the server runs the AI ​​model again to update the document.

[1054] Step 11:

[1055] The server resends the corrected document to the terminal. The input data is the corrected document. Specifically, the server sends the data to the terminal using network communication. The output is the terminal receiving the corrected document.

[1056] Step 12:

[1057] The terminal redisplays the revised document to the user. The input data is the revised document. Specifically, the terminal uses its screen display function to visually present the document to the user again. The output is that the revised document is made viewable by the user.

[1058] In this way, the system can efficiently perform tasks while taking the user's emotional state into consideration.

[1059] (Application Example 2)

[1060] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1061] Traditional personal agent systems provided the functionality to automate tasks based on user behavior patterns and preferences, but they did not take user emotions into consideration. As a result, users may experience stress or dissatisfaction. Especially in physical stores, responding to customers' emotional states is crucial, but this was difficult with current systems. Therefore, there was a need for technology that could provide more appropriate and personalized customer service based on customer emotions.

[1062] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1063] In this invention, the server includes means for collecting user data, means for analyzing the collected user data and emotional data to learn the user's behavioral patterns, preferences, and emotional states, and means for automatically executing user tasks based on the learned behavioral patterns, preferences, and emotional states. This makes it possible to detect the customer's emotional state in real time and provide optimal advice accordingly.

[1064] "User data" refers to all information about a user, including their behavior, preferences, instructions, and feedback.

[1065] "Emotional data" refers to information about a user's emotional state, analyzed from their facial expressions, voice, text, etc.

[1066] "Analysis" refers to the process of evaluating and classifying collected data to extract meaningful information.

[1067] "Behavioral patterns" refer to the tendencies of actions and habits that users repeatedly perform.

[1068] "Preferences" refer to a user's likes and dislikes, or their tendencies to prefer certain things.

[1069] "Learning" refers to the process where an algorithm uses data to build a model of user behavior patterns and preferences, and then uses that model for prediction and decision-making.

[1070] A "task" refers to a series of actions or procedures performed based on instructions from a user.

[1071] "Execute automatically" means that the system performs a pre-configured process without user intervention.

[1072] "Terminal" refers to information processing devices such as computers, smartphones, and tablets used by users.

[1073] "Feedback" refers to evaluations and opinions provided by users regarding the results.

[1074] "Correction" refers to making improvements or changes to the initial results based on the feedback provided.

[1075] "Detecting customer emotional states in real time" means recognizing customer emotions instantly, without any time lag.

[1076] "Advice" refers to the recommended actions or instructions provided by the system.

[1077] This invention realizes a system that combines an emotion engine with a user's personal agent to analyze the emotional state of the user and customer and provide optimal advice accordingly. Each part of the system operates in response to the server, terminal, and user.

[1078] System Overview

[1079] The server has the capability to collect and analyze user data and sentiment data. This utilizes natural language processing techniques and sentiment recognition algorithms. Specifically, software such as TensorFlow, OpenCV, and SpeechRecognition are used. The collected data is used to learn the user's behavioral patterns, preferences, and emotional states using machine learning algorithms. The server also automatically performs user tasks based on the learned model and sends the results to the user's device.

[1080] The device displays results sent from the server to the user and receives real-time feedback. It also plays a role in transferring the user's emotional data to the server. Smart glasses are equipped with a camera and microphone, which can capture the user's (or customer's) facial expressions and voice. This allows for real-time analysis of customer emotions and provides staff with the most appropriate advice.

[1081] Program Processing Overview

[1082] The server collects and analyzes user data and identifies emotional data using an emotion engine. For example, it can analyze emotions from a user's text, voice, and facial expressions, and recognize their emotional state in real time. The server also uses machine learning algorithms to learn user behavior patterns and preferences from the collected data. Based on this, it automatically executes tasks and sends the results to the terminal.

[1083] The terminal displays the results sent from the server to the user and receives user feedback. The feedback is sent to the server for further analysis and correction. The corrected results are then sent back to the terminal and displayed to the user.

[1084] As a concrete example, consider the following scenario: While a customer is examining a product, a staff member wears smart glasses. The system analyzes the customer's facial expression, and if it detects a satisfied expression, the staff member's glasses display the advice, "The customer is happy. Let's suggest additional products." In another scenario, if the customer appears confused, the system provides the advice, "The customer is confused. Let's provide a more detailed explanation."

[1085] An example of a specific prompt for a generative AI model is: "Please provide a prompt for a generative AI model to recognize a customer's smile and advise on how to respond if the customer is happy." This will enable the AI ​​model to automatically generate appropriate advice.

[1086] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1087] Step 1:

[1088] The server collects user and sentiment data. Specifically, it collects data such as user and customer behavior, preferences, facial expressions, voice, and text. This input data is sent to the server and stored in a database. This data collection provides initial information for use throughout the system.

[1089] Step 2:

[1090] The server analyzes the collected user data and emotion data. Natural language processing techniques and emotion recognition algorithms are used in this process. Specifically, machine learning libraries such as Python's TensorFlow are used to analyze text data, and OpenCV is used to analyze facial expression data. The analysis results are extracted as user behavior patterns, preferences, and emotional states.

[1091] Step 3:

[1092] Based on the analysis results, the server uses machine learning algorithms to model and learn user behavior patterns, preferences, and emotional states. This generates a predictive model for future task performance. The model's accuracy improves through sequential feedback during this learning process.

[1093] Step 4:

[1094] The user or device instructs the server to perform a specific task. For example, it might instruct the server to "prepare the meeting materials for next week." This instruction includes information such as the date and time, format, and a summary of the content, as well as the user's current emotional state. This becomes the input sent to the server.

[1095] Step 5:

[1096] The server automatically performs tasks requested by the user based on a trained model. For example, if the user is stressed, it automatically generates simple and easy-to-understand meeting materials. In this process, it uses a generative AI model to select and process appropriate templates and content, and then outputs the final materials.

[1097] Step 6:

[1098] The server sends the task results to the terminal. The generated materials and advice are delivered to the user's terminal and displayed to the user. At this point, output to the user is complete, and the user can review the results as needed.

[1099] Step 7:

[1100] Users provide feedback on the results displayed on their devices. For example, they can send detailed requests to the server via their devices, such as "Please describe this part in more detail." This becomes the user's input data.

[1101] Step 8:

[1102] The server analyzes the received feedback and makes necessary corrections. Specifically, it uses machine learning algorithms again to adjust the task results, taking the feedback into consideration, and makes appropriate corrections. The corrected results are then sent back to the terminal.

[1103] Step 9:

[1104] The device then redisplays the corrected results to the user. This is where final confirmation takes place, and the user can provide further feedback on the results. This ensures that the optimal result, tailored to the user's requirements down to the smallest detail, is achieved.

[1105] Specific actions

[1106] The smart glasses capture the customer's facial expressions and voice in real time using their camera and microphone.

[1107] The acquired data is sent to a server and analyzed using natural language processing technology and emotion recognition algorithms.

[1108] If the customer is happy, advise the staff, "The customer is happy. Let's suggest additional products."

[1109] If the customer is confused, advise them to "The customer is confused. Let's provide a detailed explanation."

[1110] An example of a specific prompt for a generative AI model is: "Please provide a prompt for a generative AI model to recognize a customer's smile and advise on how to respond if the customer is happy." This will enable the AI ​​model to automatically generate appropriate advice.

[1111] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1112] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1113] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1114] [Fourth Embodiment]

[1115] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1116] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1117] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1118] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1119] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1120] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1121] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1122] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1123] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1124] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1125] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1126] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1127] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1128] System Overview

[1129] The present invention is a system that efficiently performs user tasks using the user's personal agent, and specifically includes processes for collecting, analyzing, learning, automatically executing tasks, providing results, receiving feedback, and making corrections based on user data.

[1130] Program Processing Overview

[1131] Learning Module

[1132] The server first collects user data related to the user's behavior and preferences. This includes, for example, the user's past email content, web browsing history, and purchase history. Then, this data is analyzed using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences. This builds a model for predicting the user's future behavior. The model is stored on the server and made available to the agent in real time.

[1133] Task execution module

[1134] The terminal forwards user instructions to the server. For example, if a user instructs the server to "create meeting materials for next week," the server receives the instruction along with the necessary information (date and time, format, and summary of content). The server selects the optimal template based on learned behavioral patterns and preferences and automatically executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are then sent from the server to the terminal.

[1135] Interface module

[1136] The terminal displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part to be described in more detail," that feedback is sent to the server via the terminal. The server analyzes the feedback and makes any necessary corrections. The corrected results are then sent back to the terminal and displayed to the user.

[1137] Explanation with specific examples

[1138] For preparing meeting materials for next week

[1139] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[1140] 2. The terminal forwards the instructions to the server, which then selects an appropriate template by referring to past meeting materials and user preferences.

[1141] 3. The server automatically generates meeting materials based on the template.

[1142] 4. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[1143] 5. The user reviews the document and enters feedback into the device, such as "I would like this part added."

[1144] 6. The terminal forwards the feedback to the server, which analyzes the feedback and corrects the document.

[1145] 7. The revised document is sent back to the terminal and displayed to the user.

[1146] This frees users from the burden of document creation, allowing them to efficiently focus on important tasks. By having agents automatically perform tasks, users can save significant time and effort.

[1147] The following describes the processing flow.

[1148] Step 1:

[1149] The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[1150] Step 2:

[1151] The terminal receives user instructions and transmits those instructions along with necessary information (such as the date, time, format, and summary of the meeting) to the server.

[1152] Step 3:

[1153] The server analyzes the received instructions and selects the most suitable template based on the user's past data (past meeting materials, emails, preference data).

[1154] Step 4:

[1155] The server automatically generates meeting materials by filling in the necessary content based on the selected template.

[1156] Step 5:

[1157] The server transfers the generated meeting materials to the terminal.

[1158] Step 6:

[1159] The terminal displays the meeting materials it has received to the user.

[1160] Step 7:

[1161] The user reviews the displayed meeting materials and enters any necessary feedback into the device, such as "Please elaborate on this section."

[1162] Step 8:

[1163] The device receives user feedback and forwards it to the server.

[1164] Step 9:

[1165] The server analyzes the feedback received and makes revisions to the meeting materials.

[1166] Step 10:

[1167] The server resends the revised meeting materials to the terminal.

[1168] Step 11:

[1169] The terminal displays the revised meeting materials that have been resent to the user.

[1170] Step 12:

[1171] The user performs a final review and provides feedback again if necessary. This cycle continues until the document is finally finalized.

[1172] (Example 1)

[1173] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1174] Traditional task management systems were inefficient because users had to manually set tasks, check the results, and make corrections if necessary. Furthermore, it was difficult to optimize task execution based on individual user preferences and behavioral patterns. Therefore, there was a need for a system that would reduce the burden on users and allow for more efficient task execution.

[1175] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1176] In this invention, the server includes means for collecting user data, means for analyzing the collected user data using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences, means for automatically executing the user's tasks based on the learned behavior patterns and preferences, selecting the optimal template and generating results, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, and means for analyzing the received feedback and modifying the task results. This enables the automation of tasks based on the user's individual preferences and behavior patterns, and adaptive modification based on feedback.

[1177] "User data" refers to information related to a user's behavior and preferences, such as the content of emails, web browsing history, and purchase history.

[1178] "Natural language processing technology" refers to the technology that uses computers to understand, generate, and analyze human language, such as text and audio data.

[1179] "Clustering techniques" refer to analytical methods that group similar data together to understand the structure of the data.

[1180] "Behavioral patterns" refer to the tendencies of a series of actions and operations that a user has performed in the past.

[1181] "Preferences" refer to the individual user's tastes and preferences.

[1182] A "template" refers to a pre-prepared format or template for performing a specific task.

[1183] "Feedback" refers to comments and suggestions for improvements provided by users regarding the results.

[1184] A "machine learning algorithm" refers to a mathematical method used to learn patterns based on data and perform tasks such as prediction and classification.

[1185] Modes for carrying out the invention

[1186] This invention provides a system for collecting, analyzing, learning from, and automatically executing user data, thereby reducing the burden on users and efficiently carrying out tasks. The specific configuration and operation of this system are described below.

[1187] System Configuration

[1188] This system consists of a server for collecting and analyzing user data, and terminals where users input instructions. The server is a high-performance computer system that uses databases, natural language processing techniques (e.g., BERT and GPT-3), clustering methods (e.g., K-means), and machine learning algorithms. The terminals are devices that users can directly operate (e.g., PCs, smartphones, tablets).

[1189] Program Processing Overview

[1190] The server first collects data related to user behavior and preferences. Specifically, it obtains data through email servers, browser logs, and APIs of online shopping sites. This data includes past email content, web browsing history, and purchase history.

[1191] Next, the server analyzes the collected data using natural language processing techniques (e.g., BERT, GPT-3) and clustering methods (e.g., K-means). Through this analysis, user behavior patterns and preferences are learned, and individual user models are constructed.

[1192] When a user enters a task into the terminal, the terminal forwards this instruction to the server. For example, if a user gives the instruction "Prepare the meeting materials for next week," the server will send the instruction along with the necessary information (date and time, format, and summary of content).

[1193] The server selects the optimal template based on a trained user model and executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are sent from the server to the terminal, which then displays the materials to the user.

[1194] When a user reviews a document and provides feedback as needed, the device forwards this feedback to the server. The server analyzes the feedback and makes any necessary corrections. The corrected document is then sent back to the device and displayed to the user.

[1195] Specific example

[1196] For example, in a task to create meeting materials for the following week, the user inputs the instruction "Create meeting materials for next week" using their device. The device forwards this instruction to the server, which selects an appropriate template by referring to past meeting materials and the user's preferences. Based on this template, the server automatically generates the meeting materials and sends them to the device. When the user reviews the materials and provides feedback such as "I would like this section to be explained in more detail," the server analyzes the feedback and revises the materials. The revised materials are then sent to the device and displayed to the user.

[1197] Example of a prompt

[1198] An example of a prompt message is, "GPT-3, if the user asks you to create meeting materials for next week, generate new materials using past materials as a reference." By using prompts like this, the generative AI model can appropriately perform the task requested by the user.

[1199] As described above, the system of the present invention can automatically perform tasks based on the user's preferences and behavioral patterns, and can make adaptive modifications based on feedback.

[1200] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1201] Step 1: Collecting User Data

[1202] The server collects data related to the user's past behavior and preferences.

[1203] Specific operation: The server connects to the mail server API and retrieves email data from the past six months. It also collects browser browsing history data from log files and retrieves purchase history from shopping sites via API.

[1204] Input: Data from mail servers, browser log files, and shopping site APIs.

[1205] Output: The user's email content, web browsing history, and purchase history are stored in the server's database.

[1206] Step 2: Analyze and learn from user data

[1207] The server analyzes the collected data and learns the user's behavior patterns and preferences.

[1208] Specific operation: The server uses a natural language processing library (e.g., BERT, GPT-3) to extract important keywords from email content. Next, it applies a clustering algorithm (e.g., K-means) to classify web browsing history into different categories.

[1209] Input: User data collected in Step 1.

[1210] Output: The analyzed behavioral patterns and preferences are saved as a user model.

[1211] Step 3: Receiving the Task

[1212] The user enters the task details via their terminal. The task is then transferred to the server.

[1213] Specific action: The user enters "Create meeting materials for next week" into the application's form and presses the submit button. The device sends this data to the server as an HTTP request.

[1214] Input: Task instructions entered by the user on the device.

[1215] Output: The task instruction reaches the server and is added to the processing queue.

[1216] Step 4: Task execution and result generation

[1217] The server executes tasks and generates results based on the user's behavior patterns and preferences.

[1218] Specific operation: The server selects the most suitable meeting material template from user preference data and automatically generates new materials based on past meeting material data. It adds necessary context and content using a generation AI model (e.g., GPT-3).

[1219] Input: User model, past meeting materials, templates.

[1220] Output: The generated meeting materials file is saved to the server.

[1221] Step 5: Providing Results

[1222] The terminal displays the results received from the server to the user.

[1223] Specific operation: The server sends the generated meeting materials file to the terminal, and the terminal's application displays this file in the user interface.

[1224] Input: Meeting materials file sent from the server.

[1225] Output: The user is able to view the meeting materials on their device.

[1226] Step 6: Receiving Feedback

[1227] Users review the materials and provide feedback.

[1228] Specific action: The user enters feedback into a form on their device, stating "I would like this part described in more detail," and presses the submit button. The device then sends this feedback to the server.

[1229] Input: Feedback entered by the user on the device.

[1230] Output: Feedback reaches the server and is analyzed.

[1231] Step 7: Corrections based on feedback

[1232] The server analyzes the feedback it receives and corrects the document.

[1233] Specific operation: The server analyzes the feedback using a natural language processing tool and makes necessary corrections using a generation AI model. The corrected document is then regenerated and sent to the terminal.

[1234] Input: User feedback.

[1235] Output: The revised meeting materials are sent to the terminal and displayed to the user again.

[1236] The above outlines the specific processing steps of the system. This series of steps enables automated task execution and adaptive correction, significantly reducing the burden on the user.

[1237] (Application Example 1)

[1238] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1239] In modern society, the series of tasks and payment management that users perform daily have become increasingly complex, and optimization is needed. In particular, there is a lack of efficient and secure payment methods that suggest the optimal payment method based on the user's purchase history and transaction patterns, which often leads to users wasting time and effort. To address this challenge, a system is needed that collects and analyzes user data and uses trained models to efficiently automate tasks.

[1240] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1241] In this invention, the server includes means for collecting user data, means for analyzing the collected user data and learning the user's behavior patterns and preferences, means for automatically executing the user's tasks based on the learned behavior patterns and preferences, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, means for modifying the task results based on the received feedback, means for analyzing the user's purchase history and transaction patterns and suggesting the optimal payment method, means for generating reminder notifications based on the payment method, and means for quickly processing payments on the user's terminal. This enables efficient and secure task management and payment processing for users, saving them time and effort.

[1242] "User data" refers to various types of information related to a user, such as their purchase history, transaction patterns, and preferences.

[1243] "Analysis" refers to the process of analyzing collected user data to reveal user behavior patterns and preferences.

[1244] "Behavioral patterns" refer to the tendencies and habits of how a user behaves in specific situations.

[1245] "Preferences" refer to the user's tendency to choose things and services that they particularly like.

[1246] A "task" refers to a specific task or operation that a user is required to perform.

[1247] "Device" refers to an electronic device used by a user, and includes smartphones, computers, and other similar devices.

[1248] "Feedback" refers to opinions, evaluations, and requests for corrections provided by users.

[1249] "Modification" refers to the act of changing task results, settings, etc., based on user feedback.

[1250] "Purchase history" refers to information about the products and services that a user has purchased in the past.

[1251] "Trading patterns" refer to the tendencies and habits of a user in their trading.

[1252] "Payment method" refers to the payment method selected by the user, and includes credit cards, debit cards, electronic money, etc.

[1253] A "reminder notification" refers to an automated notification that informs the user of specific tasks, payment deadlines, or other important information.

[1254] "Payment processing" refers to the act of actually completing a monetary transaction using the selected payment method.

[1255] Modes for carrying out the invention

[1256] System Overview

[1257] This invention is a system that efficiently performs user tasks using a user's personal agent. Specifically, it includes processes for collecting, analyzing, and learning user data, automatically executing tasks, providing results, receiving feedback, and making corrections. It also includes analyzing the user's purchase history and transaction patterns, suggesting the optimal payment method, generating payment reminder notifications, and processing them quickly on the terminal.

[1258] Program Processing Overview

[1259] Learning Module

[1260] The server first collects user data related to the user's behavior and preferences. This includes the user's past email content, web browsing history, and purchase history. Next, the collected data is analyzed using natural language processing techniques and clustering methods to learn the user's behavior patterns and preferences. Using these learning results, a model is built to predict the user's future behavior. The model is stored on the server and made available to the agent in real time.

[1261] Task execution module

[1262] The terminal forwards user instructions to the server. For example, if a user instructs the server to "create meeting materials for next week," the server receives the instruction along with the necessary information (date and time, format, and summary of content). The server selects the optimal template based on learned behavioral patterns and preferences and automatically executes the task. Specifically, it automatically generates meeting materials by referring to past meeting materials and user preferences. The generated results are then sent from the server to the terminal.

[1263] Interface module

[1264] The terminal displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user enters feedback such as "I would like this part described in more detail" into the terminal, that feedback is sent to the server. The server analyzes the feedback, makes any necessary corrections, and sends the corrected results back to the terminal.

[1265] Payment management module

[1266] The server collects and analyzes the user's purchase history and transaction patterns to suggest the most suitable payment method. For example, it suggests the best payment method from options such as credit cards, debit cards, and electronic money based on the user's past usage. Furthermore, if a payment deadline is approaching, it generates a reminder notification and automatically sends it to the user's device.

[1267] Specific example

[1268] For preparing meeting materials for next week

[1269] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[1270] 2. The terminal forwards the instructions to the server, which then selects an appropriate template by referring to past meeting materials and user preferences.

[1271] 3. The server automatically generates meeting materials based on the template.

[1272] 4. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[1273] 5. The user reviews the document and enters feedback into the device, such as "I would like this part added."

[1274] 6. The terminal forwards the feedback to the server, which analyzes the feedback and corrects the document.

[1275] 7. The revised document is sent back to the terminal and displayed to the user.

[1276] In the case of payment management

[1277] 1. The server collects and analyzes the user's purchase history and transaction patterns.

[1278] 2. Based on the analysis results, the server suggests the most suitable payment method to the user.

[1279] 3. The user selects a suggested payment method and completes the payment on the terminal.

[1280] 4. When the payment deadline approaches, the server generates a reminder notification and sends it to the user's device.

[1281] Example of a prompt

[1282] Suggest the best payment method for user ID 12345, based on their first-quarter purchase history data.

[1283] This system frees users from the burden of creating documents and allows them to efficiently and securely select and manage payment methods.

[1284] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1285] Step 1:

[1286] The server collects user data (purchase history, transaction patterns, preferences, etc.). Specifically, it retrieves information such as the user's past email content, web browsing history, and purchase history from a database. This allows for the systematic accumulation of user data.

[1287] Step 2:

[1288] The server analyzes the collected user data. It uses natural language processing techniques to analyze the data and learn user behavior patterns and preferences. Specifically, it tokenizes text data and performs sentiment analysis and relationship extraction. It uses clustering techniques to classify the data into groups. This allows it to build a model for predicting future user behavior.

[1289] Step 3:

[1290] The server automatically performs tasks based on the learning results. Specifically, when a user gives an instruction such as "Create meeting materials for next week," the necessary information (date and time, format, and summary of content) is entered along with the instruction. The server refers to the learned model, automatically selects the optimal template, and generates the meeting materials using historical data.

[1291] Step 4:

[1292] The server sends the results of the executed task to the terminal. The generated meeting materials are sent from the server to the terminal and displayed to the user. This allows the user to review the generated materials.

[1293] Step 5:

[1294] Users review the generated documents and provide feedback. They input feedback, such as "I'd like this section described in more detail," through their device, and this feedback information is sent to the server.

[1295] Step 6:

[1296] The server analyzes the received feedback and corrects the task results. Specifically, based on the feedback, it uses natural language processing techniques to modify and add to the text, updating the document. A newly corrected document is then generated.

[1297] Step 7:

[1298] The revised document is sent again from the server to the terminal and displayed to the user. The user can then review the revised document.

[1299] Step 8:

[1300] The server analyzes the user's purchase history and transaction patterns to suggest the optimal payment method. Specifically, it uses clustering techniques and other machine learning algorithms to determine the best payment method (credit card, debit card, e-money, etc.) for the user.

[1301] Step 9:

[1302] The server sends payment methods to the user's terminal and displays them to the user. The user reviews the suggestions and selects a payment method.

[1303] Step 10:

[1304] The payment is processed on the terminal based on the payment method selected by the user. Specifically, the payment process is initiated and the payment is completed.

[1305] Step 11:

[1306] As the payment deadline approaches, the server automatically generates a reminder notification and sends it to the user's device. This helps users remember to make their payments.

[1307] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1308] System Overview

[1309] The present invention provides a system that efficiently performs user tasks using a user's personal agent, and in particular, offers more personalized assistance by combining it with an emotion engine that recognizes the user's emotions. The system includes processes for collecting, analyzing, learning from, automatically executing tasks, providing results, receiving feedback, and making corrections to user data.

[1310] Program Processing Overview

[1311] Introducing an emotional engine

[1312] The server collects data related to user behavior and preferences, as well as data to recognize the user's emotional state. Specifically, it analyzes emotions from the user's text, voice, and facial expressions, and uses an emotion engine to recognize this data as emotions. For example, it analyzes the user's facial expressions and tone of voice while they are reading an email to determine their stress level, joy, sadness, etc.

[1313] Learning Module

[1314] The server analyzes the collected user data and sentiment data. This allows it to learn user behavior patterns, preferences, and emotional states. Natural language processing techniques, clustering methods, and sentiment recognition algorithms are used in the analysis. Based on the obtained information, a model is built to address the user's emotional state. This model is stored on the server and made available to the agent in real time.

[1315] Task execution module

[1316] The terminal forwards user instructions to the server. If the user instructs, "Prepare meeting materials for next week," the server receives the instruction along with necessary information (date, time, format, summary of content) and the user's current emotional state. Based on learned behavioral patterns, preferences, and emotional states, the server selects the most suitable template and automatically executes the task. For example, if the user is stressed, the server may generate more concise and simpler materials. The generated results are then sent from the server back to the terminal.

[1317] Interface module

[1318] The device displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part to be described in more detail," that feedback is sent to the server via the device. The server analyzes the feedback and the user's emotional state and makes any necessary corrections. The corrected results are then sent back to the device and displayed to the user.

[1319] Explanation with specific examples

[1320] For preparing meeting materials for next week

[1321] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[1322] 2. The device transmits instructions, necessary information, and user sentiment data to the server.

[1323] 3. The server references the user's past data and emotional state and selects an appropriate template.

[1324] 4. The server automatically generates meeting materials based on templates. For example, if a user is feeling stressed, it will generate a document that summarizes the information concisely.

[1325] 5. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[1326] 6. Users review the materials and provide feedback as needed. For example, they might enter feedback such as, "I would like this section to be explained in more detail."

[1327] 7. The device transfers feedback and sentiment data to the server.

[1328] 8. The server analyzes feedback and sentiment data and modifies the document.

[1329] 9. The revised document is sent back to the terminal and displayed to the user.

[1330] This system not only frees users from the burden of document creation but also allows them to receive optimal support tailored to their emotional state. Agents automatically perform tasks and respond to the user's emotional state, significantly saving time and effort.

[1331] The following describes the processing flow.

[1332] Step 1:

[1333] The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[1334] Step 2:

[1335] The device receives user instructions and simultaneously collects user emotional data (text, voice, facial expressions, etc.).

[1336] Step 3:

[1337] The device analyzes the user's emotional data and uses an emotion engine to recognize the user's current emotional state.

[1338] Step 4:

[1339] The terminal transmits the instructions, necessary information (date, time, format, and summary of the meeting), and emotional state to the server.

[1340] Step 5:

[1341] The server analyzes the instructions and emotional state received, and selects the optimal template based on the user's past data (past meeting materials, emails, preference data).

[1342] Step 6:

[1343] The server automatically fills in the necessary content based on the selected template and generates meeting materials. For example, if the user is under stress, it will create materials that are concise and easy to understand visually.

[1344] Step 7:

[1345] The server transfers the generated meeting materials to the terminal.

[1346] Step 8:

[1347] The terminal displays the meeting materials it has received to the user.

[1348] Step 9:

[1349] The user reviews the displayed meeting materials and enters any necessary feedback into the device, such as "Please elaborate on this section."

[1350] Step 10:

[1351] The device then collects user feedback and their emotional state at that time, and transmits it to the server.

[1352] Step 11:

[1353] The server analyzes the feedback and emotional state received and revises the meeting materials accordingly. For example, if the user is in an optimistic emotional state, it creates more comprehensive materials that include additional details.

[1354] Step 12:

[1355] The server resends the revised meeting materials to the terminal.

[1356] Step 13:

[1357] The terminal displays the revised meeting materials that have been resent to the user.

[1358] Step 14:

[1359] The user performs a final review and provides feedback again if necessary. This cycle continues until the document is finally finalized.

[1360] (Example 2)

[1361] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1362] Traditional personal agents could automate tasks based on user behavior patterns and preferences, but they could not consider the user's emotional state. As a result, they could not provide appropriate support depending on the user's emotional state, and in some cases, this could increase the user's stress. Furthermore, the automated execution of tasks and the incorporation of feedback often did not yield optimal results because they could not consider the emotional state. Therefore, there is a need for improved user satisfaction and efficient task execution.

[1363] In Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting user data, means for recognizing the user's emotional state using an emotion engine, means for analyzing the collected user data and emotional data to learn the user's behavioral patterns, preferences, and emotional state, means for automatically executing the user's tasks based on the learned behavioral patterns, preferences, and emotional state, means for transmitting the results of the executed tasks to the user's terminal, means for receiving user feedback, and means for modifying the task results based on the received feedback and emotional data. This enables personalized support that corresponds to the user's emotional state, allowing for efficient task execution and improved user satisfaction.

[1364] "User data" refers to information such as user behavior, preferences, and instructions.

[1365] "Emotional data" refers to information about a user's emotional state extracted from their facial expressions, tone of voice, and text.

[1366] An "emotion engine" refers to a technology or software used to analyze and recognize a user's emotional state.

[1367] "Natural language processing technology" refers to the technology used to analyze text data and understand its meaning.

[1368] "Clustering techniques" refer to methods for grouping data according to similar characteristics.

[1369] A "generative AI model" refers to a model that uses artificial intelligence to generate text and other data.

[1370] A "machine learning algorithm" refers to an algorithm that learns from experience and recognizes patterns.

[1371] "User behavior patterns" refer to a user's tendencies to behave in specific situations.

[1372] "Preferences" refer to the things or tendencies that users like.

[1373] "Feedback" refers to opinions and requests for corrections provided by users.

[1374] This invention provides more personalized support in a system that efficiently performs user tasks using a user's personal agent, particularly by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[1375] Introducing an emotional engine

[1376] The server collects data related to user behavior and preferences, as well as data to recognize the user's emotional state. Specifically, it analyzes emotions from the user's text, voice, and facial expressions, and uses an emotion engine to recognize this data as emotions. For example, it analyzes the user's facial expressions and tone of voice while they are reading an email to determine their stress level, joy, sadness, etc.

[1377] Learning Module

[1378] The server analyzes the collected user data and sentiment data. This allows it to learn user behavior patterns, preferences, and emotional states. Natural language processing techniques, clustering methods, and sentiment recognition algorithms are used for the analysis. Based on the obtained information, a model is built to respond to the user's emotional state. This model is stored on the server and made available to the agent in real time.

[1379] Task execution module

[1380] The terminal forwards user instructions to the server. For example, if a user instructs the terminal to "create meeting materials for next week," the server receives the instruction along with necessary information (date and time, format, summary of content) and the user's current emotional state. The server selects the most suitable template based on learned behavioral patterns, preferences, and emotional states, and automatically executes the task. For example, if the user is stressed, the server may generate more concise and simpler materials. The generated results are then sent from the server back to the terminal.

[1381] Interface module

[1382] The device displays the results received from the server to the user. The user reviews the results and provides feedback as needed. For example, if the user sends feedback such as "I would like this part described in more detail," that feedback is sent to the server via the device. The server analyzes the feedback and the user's emotional state and makes any necessary corrections. The corrected results are then sent back to the device and displayed to the user.

[1383] Specific example

[1384] Next, let's consider a specific example: preparing meeting materials for next week.

[1385] 1. The user uses their device to input the instruction, "Prepare the meeting materials for next week."

[1386] 2. The device transmits instructions, necessary information, and user sentiment data to the server.

[1387] 3. The server references the user's past data and emotional state and selects an appropriate template.

[1388] 4. The server automatically generates meeting materials based on templates. For example, if a user is feeling stressed, it will generate a document that summarizes the information concisely.

[1389] 5. The server sends the generated document to the terminal, and the terminal displays the document to the user.

[1390] 6. The user reviews the document and provides feedback as needed. For example, they might enter feedback such as, "I would like this section to be explained in more detail."

[1391] 7. The device transfers feedback and sentiment data to the server.

[1392] 8. The server revises the document based on the feedback and sends it back to the terminal. The final result is displayed to the user.

[1393] This system allows users to efficiently complete tasks and receive optimal support tailored to their emotions.

[1394] Example of a prompt

[1395] "Please prepare the meeting materials for next week. Please keep the information concise and to the best of your ability, taking your stress levels into consideration."

[1396] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1397] Step 1:

[1398] The user uses a device to input an instruction, such as "Prepare the meeting materials for next week." This input includes the instruction itself and necessary related information (e.g., date and time, format, and summary of the content). The device recognizes this input and also determines the user's current emotional state (collected from facial expressions, voice, etc.). The input data includes the instruction text and emotional data. This data is temporarily stored within the device.

[1399] Step 2:

[1400] The terminal transfers collected instruction content and sentiment data to the server. Input includes instruction text, date and time, format, content summary, and sentiment data. Output after transfer is achieved when the server receives this data. Specifically, the terminal sends data to the server using network communication.

[1401] Step 3:

[1402] The server analyzes the received instruction text and sentiment data. Natural language processing techniques are used for the analysis. The input data consists of instruction text and sentiment data. The analysis results in an understanding of the instruction content and a determination of the emotional state. The output includes the analysis results and the user's emotional state. Specifically, the server executes a text analysis engine and an sentiment recognition algorithm.

[1403] Step 4:

[1404] The server selects an appropriate template based on a model that has learned past behavioral patterns, preferences, and emotional states. Input data includes analysis results and historical data. This data is evaluated using clustering techniques to determine the optimal template. The output is the selected template. Specifically, the server retrieves historical data from the database, applies the machine learning model, and selects a template.

[1405] Step 5:

[1406] The server creates documents using the selected template. A generative AI model is used here. The input data includes the template and the necessary information. Since the generated documents also take emotional states into account, for example, a concise document will be generated for a user experiencing stress. The output is the generated document. Specifically, the server runs the generative AI model and embeds information into the template.

[1407] Step 6:

[1408] The server sends the generated document to the terminal. The data sent is the generated document. Specifically, the server sends the data to the terminal using network communication. The output is the terminal receiving the document.

[1409] Step 7:

[1410] The terminal displays the received materials to the user. The input data is generated material sent from the server. Specifically, the terminal uses its screen display function to visually provide the materials to the user. The output is that the materials become viewable by the user.

[1411] Step 8:

[1412] The user reviews the document and enters feedback. The input data consists of user correction instructions and additional information. Specifically, the user enters text using a terminal. The output is the feedback text.

[1413] Step 9:

[1414] The terminal retransmits the input feedback and sentiment data to the server. The input data consists of feedback text and sentiment data. Specifically, the terminal sends data to the server using network communication. The output is the server receiving this data again.

[1415] Step 10:

[1416] The server analyzes user feedback and sentiment data to revise the document. Input data consists of feedback text and sentiment data. The analysis determines the necessary revisions to the document. The output is the revised document. Specifically, the server runs the AI ​​model again to update the document.

[1417] Step 11:

[1418] The server resends the corrected document to the terminal. The input data is the corrected document. Specifically, the server sends the data to the terminal using network communication. The output is the terminal receiving the corrected document.

[1419] Step 12:

[1420] The terminal redisplays the revised document to the user. The input data is the revised document. Specifically, the terminal uses its screen display function to visually present the document to the user again. The output is that the revised document is made viewable by the user.

[1421] In this way, the system can efficiently perform tasks while taking the user's emotional state into consideration.

[1422] (Application Example 2)

[1423] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1424] Traditional personal agent systems provided the functionality to automate tasks based on user behavior patterns and preferences, but they did not take user emotions into consideration. As a result, users may experience stress or dissatisfaction. Especially in physical stores, responding to customers' emotional states is crucial, but this was difficult with current systems. Therefore, there was a need for technology that could provide more appropriate and personalized customer service based on customer emotions.

[1425] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1426] In this invention, the server includes means for collecting user data, means for analyzing the collected user data and emotional data to learn the user's behavioral patterns, preferences, and emotional states, and means for automatically executing user tasks based on the learned behavioral patterns, preferences, and emotional states. This makes it possible to detect the customer's emotional state in real time and provide optimal advice accordingly.

[1427] "User data" refers to all information about a user, including their behavior, preferences, instructions, and feedback.

[1428] "Emotional data" refers to information about a user's emotional state, analyzed from their facial expressions, voice, text, etc.

[1429] "Analysis" refers to the process of evaluating and classifying collected data to extract meaningful information.

[1430] "Behavioral patterns" refer to the tendencies of actions and habits that users repeatedly perform.

[1431] "Preferences" refer to a user's likes and dislikes, or their tendencies to prefer certain things.

[1432] "Learning" refers to the process where an algorithm uses data to build a model of user behavior patterns and preferences, and then uses that model for prediction and decision-making.

[1433] A "task" refers to a series of actions or procedures performed based on instructions from a user.

[1434] "Execute automatically" means that the system performs a pre-configured process without user intervention.

[1435] "Terminal" refers to information processing devices such as computers, smartphones, and tablets used by users.

[1436] "Feedback" refers to evaluations and opinions provided by users regarding the results.

[1437] "Correction" refers to making improvements or changes to the initial results based on the feedback provided.

[1438] "Detecting customer emotional states in real time" means recognizing customer emotions instantly, without any time lag.

[1439] "Advice" refers to the recommended actions or instructions provided by the system.

[1440] This invention realizes a system that combines an emotion engine with a user's personal agent to analyze the emotional state of the user and customer and provide optimal advice accordingly. Each part of the system operates in response to the server, terminal, and user.

[1441] System Overview

[1442] The server has the capability to collect and analyze user data and sentiment data. This utilizes natural language processing techniques and sentiment recognition algorithms. Specifically, software such as TensorFlow, OpenCV, and SpeechRecognition are used. The collected data is used to learn the user's behavioral patterns, preferences, and emotional states using machine learning algorithms. The server also automatically performs user tasks based on the learned model and sends the results to the user's device.

[1443] The device displays results sent from the server to the user and receives real-time feedback. It also plays a role in transferring the user's emotional data to the server. Smart glasses are equipped with a camera and microphone, which can capture the user's (or customer's) facial expressions and voice. This allows for real-time analysis of customer emotions and provides staff with the most appropriate advice.

[1444] Program Processing Overview

[1445] The server collects and analyzes user data and identifies emotional data using an emotion engine. For example, it can analyze emotions from a user's text, voice, and facial expressions, and recognize their emotional state in real time. The server also uses machine learning algorithms to learn user behavior patterns and preferences from the collected data. Based on this, it automatically executes tasks and sends the results to the terminal.

[1446] The terminal displays the results sent from the server to the user and receives user feedback. The feedback is sent to the server for further analysis and correction. The corrected results are then sent back to the terminal and displayed to the user.

[1447] As a concrete example, consider the following scenario: While a customer is examining a product, a staff member wears smart glasses. The system analyzes the customer's facial expression, and if it detects a satisfied expression, the staff member's glasses display the advice, "The customer is happy. Let's suggest additional products." In another scenario, if the customer appears confused, the system provides the advice, "The customer is confused. Let's provide a more detailed explanation."

[1448] An example of a specific prompt for a generative AI model is: "Please provide a prompt for a generative AI model to recognize a customer's smile and advise on how to respond if the customer is happy." This will enable the AI ​​model to automatically generate appropriate advice.

[1449] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1450] Step 1:

[1451] The server collects user and sentiment data. Specifically, it collects data such as user and customer behavior, preferences, facial expressions, voice, and text. This input data is sent to the server and stored in a database. This data collection provides initial information for use throughout the system.

[1452] Step 2:

[1453] The server analyzes the collected user data and emotion data. Natural language processing techniques and emotion recognition algorithms are used in this process. Specifically, machine learning libraries such as Python's TensorFlow are used to analyze text data, and OpenCV is used to analyze facial expression data. The analysis results are extracted as user behavior patterns, preferences, and emotional states.

[1454] Step 3:

[1455] Based on the analysis results, the server uses machine learning algorithms to model and learn user behavior patterns, preferences, and emotional states. This generates a predictive model for future task performance. The model's accuracy improves through sequential feedback during this learning process.

[1456] Step 4:

[1457] The user or device instructs the server to perform a specific task. For example, it might instruct the server to "prepare the meeting materials for next week." This instruction includes information such as the date and time, format, and a summary of the content, as well as the user's current emotional state. This becomes the input sent to the server.

[1458] Step 5:

[1459] The server automatically performs tasks requested by the user based on a trained model. For example, if the user is stressed, it automatically generates simple and easy-to-understand meeting materials. In this process, it uses a generative AI model to select and process appropriate templates and content, and then outputs the final materials.

[1460] Step 6:

[1461] The server sends the task results to the terminal. The generated materials and advice are delivered to the user's terminal and displayed to the user. At this point, output to the user is complete, and the user can review the results as needed.

[1462] Step 7:

[1463] Users provide feedback on the results displayed on their devices. For example, they can send detailed requests to the server via their devices, such as "Please describe this part in more detail." This becomes the user's input data.

[1464] Step 8:

[1465] The server analyzes the received feedback and makes necessary corrections. Specifically, it uses machine learning algorithms again to adjust the task results, taking the feedback into consideration, and makes appropriate corrections. The corrected results are then sent back to the terminal.

[1466] Step 9:

[1467] The device then redisplays the corrected results to the user. This is where final confirmation takes place, and the user can provide further feedback on the results. This ensures that the optimal result, tailored to the user's requirements down to the smallest detail, is achieved.

[1468] Specific actions

[1469] The smart glasses capture the customer's facial expressions and voice in real time using their camera and microphone.

[1470] The acquired data is sent to a server and analyzed using natural language processing technology and emotion recognition algorithms.

[1471] If the customer is happy, advise the staff, "The customer is happy. Let's suggest additional products."

[1472] If the customer is confused, advise them to "The customer is confused. Let's provide a detailed explanation."

[1473] An example of a specific prompt for a generative AI model is: "Please provide a prompt for a generative AI model to recognize a customer's smile and advise on how to respond if the customer is happy." This will enable the AI ​​model to automatically generate appropriate advice.

[1474] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1475] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1476] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1477] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1478] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1479] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1480] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1481] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1482] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1483] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1484] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1485] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1486] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1487] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1488] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1489] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1490] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1491] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1492] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1493] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1494] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1495] The following is further disclosed regarding the embodiments described above.

[1496] (Claim 1)

[1497] Means of collecting user data,

[1498] A means of analyzing collected user data and learning user behavior patterns and preferences,

[1499] A means of automatically performing user tasks based on learned behavioral patterns and preferences,

[1500] A means for sending the results of the executed task to the user's terminal,

[1501] Means of receiving user feedback,

[1502] A system that includes means for correcting task results based on received feedback.

[1503] (Claim 2)

[1504] The system according to claim 1, wherein the means for analyzing the user data is natural language processing technology.

[1505] (Claim 3)

[1506] The system according to claim 1, wherein the means for automatically performing the aforementioned task is a machine learning algorithm.

[1507] "Example 1"

[1508] (Claim 1)

[1509] Means of collecting user data,

[1510] A means of learning user behavior patterns and preferences by analyzing collected user data using natural language processing techniques and clustering methods,

[1511] A means for automatically executing user tasks, selecting the optimal template, and generating results based on learned behavioral patterns and preferences,

[1512] A means for sending the results of the executed task to the user's terminal,

[1513] Means of receiving user feedback,

[1514] A system that includes means for analyzing received feedback and correcting the outcome of a task.

[1515] (Claim 2)

[1516] The system according to claim 1, wherein the user data is analyzed using natural language processing technology and clustering methods.

[1517] (Claim 3)

[1518] The system according to claim 1, wherein the means for automatically performing the aforementioned task includes means for selecting a machine learning algorithm and an optimal template to generate results.

[1519] "Application Example 1"

[1520] (Claim 1)

[1521] Means of collecting user data,

[1522] A means of analyzing collected user data and learning user behavior patterns and preferences,

[1523] A means of automatically performing user tasks based on learned behavioral patterns and preferences,

[1524] A means for sending the results of the executed task to the user's terminal,

[1525] Means of receiving user feedback,

[1526] A means of correcting the task results based on the feedback received,

[1527] A means of analyzing a user's purchase history and transaction patterns to suggest the optimal payment method,

[1528] A means of generating reminder notifications based on the payment method,

[1529] A system that includes means for quickly processing payments on the user's device.

[1530] (Claim 2)

[1531] The system according to claim 1, wherein the means for analyzing the user data is natural language processing technology.

[1532] (Claim 3)

[1533] The system according to claim 1, wherein the means for automatically performing the aforementioned task is a machine learning algorithm.

[1534] "Example 2 of combining an emotion engine"

[1535] (Claim 1)

[1536] Means of collecting user data,

[1537] A means for analyzing collected user data and emotional data to learn user behavior patterns, preferences, and emotional states,

[1538] A means for automatically performing user tasks based on learned behavioral patterns, preferences, and emotional states,

[1539] A means for sending the results of the executed task to the user's terminal,

[1540] Means of receiving user feedback,

[1541] Means for modifying task outcomes based on received feedback and sentiment data,

[1542] A means of recognizing a user's emotional state using an emotion engine,

[1543] A system that includes this.

[1544] (Claim 2)

[1545] The system according to claim 1, wherein the means for analyzing the user data and sentiment data uses natural language processing technology and clustering methods.

[1546] (Claim 3)

[1547] The system according to claim 1, wherein the means for automatically performing the aforementioned task is to use a generative AI model and a machine learning algorithm.

[1548] "Application example 2 when combining with an emotional engine"

[1549] (Claim 1)

[1550] Means of collecting user data,

[1551] A means for analyzing collected user data and emotional data to learn user behavior patterns, preferences, and emotional states,

[1552] A means for automatically performing user tasks based on learned behavioral patterns, preferences, and emotional states,

[1553] A means for sending the results of the executed task to the user's terminal,

[1554] Means of receiving user feedback,

[1555] A means of correcting the task results based on the feedback received,

[1556] A system that includes means to detect a customer's emotional state in real time and provide appropriate advice accordingly.

[1557] (Claim 2)

[1558] The system according to claim 1, wherein the means for analyzing the user data and emotion data uses natural language processing technology and an emotion recognition algorithm.

[1559] (Claim 3)

[1560] The system according to claim 1, wherein the means for automatically performing the aforementioned task is a machine learning algorithm. [Explanation of Symbols]

[1561] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means of collecting user data, A means of analyzing collected user data and learning user behavior patterns and preferences, A means of automatically performing user tasks based on learned behavioral patterns and preferences, A means for sending the results of the executed task to the user's terminal, Means of receiving user feedback, A system that includes means for correcting task results based on received feedback.

2. The system according to claim 1, wherein the means for analyzing the user data is natural language processing technology.

3. The system according to claim 1, wherein the means for automatically performing the aforementioned task is a machine learning algorithm.

Images