system

Abstract

We provide the system. [Solution] A system comprising: means for asking multiple questions to collect an individual's knowledge, thoughts, experiences, and memories; means for collecting the individual's answer data to the questions and analyzing it in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its movements, and adjusting it as necessary; means for using the avatar to perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance; and means for improving the avatar's performance based on feedback.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, it has been difficult to convert personal knowledge, experience, and thinking processes into digital data and manage them in a permanently available form. Also, in the conventional avatar technology, it has been impossible to accurately reflect personal knowledge and experience, and it has been impossible to provide the high-level consultations and business responses required by users. As a result, there has been a problem that the knowledge and experience cannot be utilized even when a person with specific expertise is absent or after death.

Means for Solving the Problems

[0005] To solve the above problems, this invention provides a system that collects an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on this data, and gives it the function of an alter ego. Specifically, it provides means for collecting knowledge and experiences from an individual through multiple questions, and for analyzing and structuring this collected data. It also provides means for generating an individual's personality model and thought patterns based on this data, and means for designing and adjusting the avatar based on the generated model. Furthermore, by providing means for the avatar to perform specific tasks according to the user's wishes, collecting execution logs to evaluate performance, and improving the avatar's performance based on feedback, the system realizes a system that can continuously utilize the user's knowledge and experience even when the user is absent or after death.

[0006] "Individual" refers to a specific person and encompasses information such as knowledge, thoughts, experiences, and memories that that person possesses.

[0007] "Knowledge" is a general term for the information, understanding, and perceptions that an individual acquires through learning and experience.

[0008] "Thinking" refers to the entire thought process that individuals engage in when solving problems or making decisions.

[0009] "Experience" includes events that an individual has experienced in the past, as well as the lessons learned and insights gained from those events.

[0010] "Memory" refers to the overall ability of an individual to store information and events acquired in the past and to recall them later.

[0011] A "question" refers to a statement made to an individual to solicit information or opinions, and is a means of collecting information through the answers received.

[0012] "Response data" refers to the content of the responses that individuals provide to questions, and analysis is performed based on this data.

[0013] "Analysis" refers to the process of examining collected data and extracting important information and patterns.

[0014] A "structured database" refers to a database that systematically organizes collected and analyzed data, and allows for efficient storage and retrieval.

[0015] A "personality model" refers to a conceptual model created based on an individual's personality and behavioral patterns.

[0016] A "thinking pattern" refers to the series of processes and methods an individual uses to process information and make decisions.

[0017] An "avatar" refers to a virtual alter ego created based on an individual's knowledge, thoughts, and experiences, and possesses the ability to perform specific tasks.

[0018] "Simulation" refers to a virtual trial used to test and verify the movements and responses of an avatar in advance.

[0019] "Operational verification" refers to the process of verifying whether the avatar functions as intended by its design.

[0020] A "specific task" refers to the concrete work or tasks that the user has their avatar perform.

[0021] An "execution log" refers to data that records the actions and responses of an avatar when it performs a specific task.

[0022] "Performance evaluation" refers to the process of assessing how well an avatar performed a specific task.

[0023] "Feedback" refers to the opinions and evaluations that users provide regarding the actions and responses of an avatar.

[0024] "Performance improvement" refers to the process of improving the functions and response accuracy of an avatar based on feedback.

Brief Explanation of Drawings

[0025] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined [Figure 14]This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0026] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0027] First, let's explain the terminology used in the following explanation.

[0028] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0029] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0030] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0031] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0033] [First Embodiment]

[0034] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0035] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0038] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0041] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0045] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0046] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. This system is implemented through the following basic phases.

[0047] 1. User data collection phase

[0048] First, the terminal asks the user an initial question. For example, it might ask, "Please tell me about your work experience." The user then enters their answer into the terminal. For example, they might answer, "I have 5 years of experience as a marketing manager." The server then analyzes the user's response data in real time and extracts important keywords and phrases. This analyzed data is immediately stored in a structured database.

[0049] Next, the server generates additional questions based on important information from the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?" The user's additional responses are also saved.

[0050] 2. Data Analysis and Processing Phase

[0051] The collected data is first cleaned by the server to remove noise and misinformation. Then, using natural language processing (NLP) techniques, the context and meaning of the user's writing are analyzed, and important decisions and thought processes are extracted. Based on this information, the server generates a personality model and thought patterns of the user. This allows for a systematic understanding of what kind of decisions the user makes in what situations.

[0052] 3. Avatar Generation Phase

[0053] The server designs an avatar based on a personality model and thought patterns it generates. This designed avatar actively utilizes the user's past data and functions as a representation of the user. Next, the terminal verifies the avatar's operation. For example, it might ask a simulated question such as, "Please explain the new market strategy." The terminal checks whether the avatar's response meets the user's expectations, and the server makes adjustments if necessary.

[0054] 4. Avatar Usage Phase

[0055] When a user is away on a business trip or too busy, the avatar can perform specific tasks on their behalf. For example, the avatar can give a presentation at a meeting as a task set by the user. The device monitors the avatar's actions and responses in real time and sends the execution logs to the server. The server analyzes these logs and evaluates the avatar's performance. Based on this evaluation, the server collects feedback from the user and improves the avatar's performance.

[0056] This allows the user's knowledge and experience to be continuously utilized even when they are absent or after their death. Furthermore, the avatar's performance can be continuously improved based on feedback, always reflecting the latest knowledge and experience.

[0057] As a concrete example, when a company leader is on a long business trip, an avatar can attend internal meetings on their behalf and make decisions based on the leader's thought process and decision-making criteria. This allows the company to maximize the use of the leader's expertise and experience even in their absence.

[0058] As described above, the present invention provides a system that can be widely applied by digitizing an individual's knowledge and experience and reproducing it as an avatar.

[0059] The following describes the processing flow.

[0060] Step 1:

[0061] The device presents the user with an initial question. For example, it might display a question such as, "Please tell us about your work experience."

[0062] Step 2:

[0063] The user enters their response into the device. For example, they might enter, "I have 5 years of experience as a marketing manager."

[0064] Step 3:

[0065] The server analyzes user response data in real time and extracts important keywords and phrases. For example, it might extract "5 years" or "marketing manager."

[0066] Step 4:

[0067] The server generates additional questions based on the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?"

[0068] Step 5:

[0069] The user enters their answer to an additional question on the device. For example, they might answer, "I led the marketing campaign for a new product."

[0070] Step 6:

[0071] The server also analyzes the additional response data, extracts important information, and saves it to the database. It also generates additional questions as needed.

[0072] Step 7:

[0073] The server cleans the collected data, removing noise and misinformation to maintain data quality.

[0074] Step 8:

[0075] The server analyzes the cleaned data using natural language processing (NLP) techniques to understand the context and meaning, and extract important decisions and thought processes.

[0076] Step 9:

[0077] The server generates a user personality model and thought patterns based on the extracted information. This allows for a systematic understanding of how users make decisions in different situations.

[0078] Step 10:

[0079] The avatar is designed based on personality models and thought patterns generated by the server. This process incorporates the user's past data.

[0080] Step 11:

[0081] The device asks the avatar simulation questions to test its functionality. For example, it might ask, "Please explain your new market strategy."

[0082] Step 12:

[0083] The terminal checks the avatar's response and verifies that it is operating according to the specifications. The server makes adjustments as needed.

[0084] Step 13:

[0085] The device performs a final confirmation with the user to verify that there are no problems with the avatar's operation. This ensures that the avatar operates according to the user's expectations.

[0086] Step 14:

[0087] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar could give a presentation at a meeting on their behalf.

[0088] Step 15:

[0089] The server collects avatar execution logs, recording the avatar's actions and responses.

[0090] Step 16:

[0091] The system analyzes the logs collected by the server to evaluate the avatar's performance. This verifies whether the avatar was able to perform the task properly.

[0092] Step 17:

[0093] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance.

[0094] Step 18:

[0095] The server periodically collects new information from users and updates avatar data to the latest version. This ensures that avatars always reflect the most up-to-date knowledge and experience.

[0096] (Example 1)

[0097] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0098] Currently, many companies and individuals are finding it difficult to digitize their knowledge and experience and utilize that data effectively. In particular, problems arise when individuals are unavailable or too busy to provide support based on their experience and knowledge, especially when making important decisions or judgments. Furthermore, removing noise and misinformation from collected data, and improving the accuracy of data analysis, remain challenges.

[0099] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0100] In this invention, the server includes means for presenting information for asking multiple questions; means for collecting and analyzing an individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for using the avatar to perform specific tasks on behalf of the individual; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for cleaning the collected data from noise and misinformation; and means for analyzing the context and meaning of the cleaned data using natural language processing technology and extracting important decisions and thought processes. This makes it possible to effectively digitize an individual's knowledge and experience and support important decision-making and judgment through the avatar. Furthermore, it allows the individual's knowledge and experience to continue to be utilized even when they are absent or busy.

[0101] An "information presentation means" is a device that presents questions or instructions to the user and collects answers or information.

[0102] "Response data" refers to the information or content that users provide or input to information presentation tools.

[0103] "Methods for real-time analysis" refers to the process and technology of immediately analyzing the content of response data as soon as it is obtained and extracting the necessary information.

[0104] A "structured database" refers to a database where the stored data is organized in a specific format so that it can be easily searched and analyzed.

[0105] A "personality model" is a data-based representation of an individual's personality and behavioral characteristics.

[0106] A "thinking pattern" is a model that extracts and represents the characteristics of an individual's judgment criteria and decision-making process.

[0107] A "simulation method" is a means of testing in advance how an avatar will actually behave.

[0108] A "proxy means" is a method by which an avatar performs a specific task on behalf of an individual.

[0109] An "execution log" refers to a detailed record of when an avatar performs a task.

[0110] "Performance evaluation" is a process that evaluates the quality of the avatar's movements and responses based on execution logs.

[0111] "Feedback" refers to evaluations and opinions regarding the results of task execution or the performance of the avatar.

[0112] "Cleaning methods" refer to techniques and processes that remove noise and misinformation from collected data, thereby improving data quality.

[0113] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human natural language.

[0114] "Important decisions and thought processes" refer to the key information and processes involved in making a particular idea or judgment.

[0115] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. This system is implemented through the following basic phases.

[0116] 1. User data collection phase

[0117] First, the terminal asks the user an initial question. For example, it might ask, "Please tell me about your work experience." The user then enters their answer into the terminal. For example, they might answer, "I have five years of experience as a marketing manager." The server then analyzes the user's response data in real time and extracts important keywords and phrases. This analyzed data is immediately stored in a structured database. Next, the server generates additional questions based on the important information from the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects did you lead?" The user's additional response data is also stored in the same way.

[0118] 2. Data Analysis and Processing Phase

[0119] The collected data is first cleaned by the server to remove noise and misinformation. Then, using natural language processing (NLP) techniques, the context and meaning of the user's writing are analyzed, and important decisions and thought processes are extracted. Based on this information, the server generates a personality model and thought patterns of the user. This allows for a systematic understanding of what kind of decisions the user makes in what situations.

[0120] 3. Avatar Generation Phase

[0121] The server designs an avatar based on a personality model and thought patterns it generates. This designed avatar actively utilizes the user's past data and functions as a representation of the user. Next, the terminal verifies the avatar's operation. For example, it might ask a simulated question such as, "Please explain the new market strategy." The terminal checks whether the avatar's response meets the user's expectations, and the server makes adjustments if necessary.

[0122] 4. Avatar Usage Phase

[0123] When a user is away on a business trip or busy, the avatar performs specific tasks on their behalf. For example, the avatar can give a presentation at a meeting as a task set by the user. The terminal monitors the avatar's actions and responses in real time and sends the execution log to the server. The server analyzes this log and evaluates the avatar's performance. Based on this evaluation, feedback from the user is collected, and the server improves the avatar's performance. This makes it possible to continuously utilize the user's knowledge and experience even when the user is absent or after their death. Furthermore, the avatar's performance continuously improves based on feedback, always reflecting the latest knowledge and experience.

[0124] Hardware and software to be used

[0125] Devices: PC, smartphone, tablet

[0126] Server: High-performance server cluster (e.g., AWS® EC2, Google® Cloud Platform)

[0127] Software technologies: Natural language processing (e.g., Spacy, BERT), database management systems (e.g., MySQL®, PostgreSQL)

[0128] Examples of specific cases and prompt statements

[0129] As a concrete example, consider a scenario where a company leader is on a long business trip, and an avatar attends internal meetings on their behalf, making decisions based on the leader's thought process and decision-making criteria. In this case, the leader can assign the following tasks to the avatar from their business trip location.

[0130] Examples of prompts to input into a generative AI model

[0131] Now, please give a presentation on our new market strategy at today's meeting. Please base your presentation on the following points:

[0132] 1. The importance and potential benefits of new markets

[0133] 2. Our strengths based on competitive analysis

[0134] 3. Specific action plan and timeline

[0135] In response to this prompt, the avatar can deliver a natural presentation based on the leader's past data and thought patterns generated by the system. The terminal monitors the avatar's actions in real time and sends feedback to the server to improve the avatar's performance.

[0136] With the above configuration, the present invention provides a system that can be widely applied by digitizing an individual's knowledge and experience and reproducing it as an avatar.

[0137] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0138] Step 1: User Data Collection Phase

[0139] Specific actions:

[0140] The device asks the user initial questions. For example, it might ask, "Please tell me about your work experience."

[0141] Input: User's response (e.g., "I have 5 years of experience as a marketing manager")

[0142] Data processing: The server analyzes this response in real time and extracts important keywords and phrases.

[0143] Output: Analyzed data (e.g., "5 years", "Marketing Manager")

[0144] Specific actions:

[0145] The server saves the analysis data to the database.

[0146] Input: Analyzed data

[0147] Output: Structured data stored in the database

[0148] Specific actions:

[0149] The server generates additional questions based on the analyzed data and sends them to the terminal.

[0150] Input: Analysis data

[0151] Data processing: Generating additional questions (e.g., "As a marketing manager, what projects did you lead?")

[0152] Output: Additional questions

[0153] Step 2: Data Analysis and Processing Phase

[0154] Specific actions:

[0155] The server cleans the collected data, removing noise and misinformation.

[0156] Input: Collected data

[0157] Data processing: Data cleaning

[0158] Output: Cleaned data

[0159] Specific actions:

[0160] The server uses NLP (Neuro-Linguistic Programming) technology to analyze context and meaning.

[0161] Input: Cleaned data

[0162] Data processing: Contextual analysis through natural language processing (e.g., using the BERT model)

[0163] Output: Information with context and meaning analyzed.

[0164] Specific actions:

[0165] The server generates a user personality model and thought patterns from the analysis results.

[0166] Input: Information analyzed for context and meaning

[0167] Data processing: Generation of personality models and thought patterns

[0168] Output: User personality model and thought patterns

[0169] Step 3: Avatar Generation Phase

[0170] Specific actions:

[0171] The avatar is designed based on personality models and thought patterns generated by the server.

[0172] Input: User's personality model and thought patterns

[0173] Data processing: Avatar design

[0174] Output: Designed avatar

[0175] Specific actions:

[0176] The device will perform a function check on the avatar.

[0177] Input: Designed avatar

[0178] Data calculation: Ask simulation questions and observe the avatar's responses (e.g., "Please explain the new market strategy").

[0179] Output: Avatar response data

[0180] Specific actions:

[0181] The server will adjust the avatar as needed.

[0182] Input: Avatar response data

[0183] Data processing: Confirmation and adjustment of responses

[0184] Output: Adjusted avatar

[0185] Step 4: Avatar Utilization Phase

[0186] Specific actions:

[0187] The avatar performs tasks set by the user.

[0188] Input: User's task settings (e.g., "Presentation at a meeting")

[0189] Data processing: Avatar performs tasks

[0190] Output: Task execution results

[0191] Specific actions:

[0192] The device monitors the avatar's movements and responses in real time and collects logs.

[0193] Input: Avatar movement data

[0194] Data processing: Collection of operation logs

[0195] Output: Collected execution logs

[0196] Specific actions:

[0197] The server analyzes the execution logs and performs a performance evaluation.

[0198] Input: Collected execution logs

[0199] Data processing: Performance evaluation

[0200] Output: Evaluation results and necessary improvements

[0201] Specific actions:

[0202] The server improves avatar performance based on user feedback.

[0203] Input: User feedback and evaluation results

[0204] Data processing: Improving avatar performance based on feedback.

[0205] Output: Improved performance avatar

[0206] (Application Example 1)

[0207] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0208] In recent years, there has been a growing demand for technology that digitizes an individual's knowledge and experience, and uses that data to generate avatars that function as digital representations. However, current technology lacks an effective means of identifying in real time what products and services a user is interested in and recommending appropriate products and services based on those preferences. Furthermore, there is a lack of means to quickly improve the avatar's behavior and responses based on user feedback. Solving these challenges is essential.

[0209] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0210] In this invention, the server includes means for asking multiple questions to collect an individual's knowledge, thoughts, experiences, and memories; means for collecting and analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for collecting the user's real-time movements and gaze via a smart terminal and analyzing the user's preferences and gaze information; and means for generating the user's avatar based on the analyzed information and recommending products and services in real time. This makes it possible to recommend products and services that reflect the user's preferences and gaze in real time.

[0211] "Personal knowledge, thoughts, experiences, and memories" refers to the collection of information, ways of thinking, actual activities and processes performed, and past experiences of a particular individual.

[0212] "Means of asking questions" refers to methods and techniques for presenting a variety of questions in order to obtain information from an individual.

[0213] "Means for collecting and analyzing response data in real time" refers to methods and technologies for quickly collecting individual responses to questions and immediately analyzing that data.

[0214] "Means for extracting important information from analyzed data and storing it in a structured database" refers to methods and techniques for analyzing collected data, identifying key elements, and storing them in a systematic database.

[0215] "Methods for generating personality models and thought patterns" refer to methods and techniques for modeling an individual's personality and thought patterns based on their behavior and responses.

[0216] "Methods for designing avatars" refers to methods and technologies for designing and creating avatars that function as extensions of an individual, based on generated personality models and thought patterns.

[0217] "Means of performing operational checks and making adjustments as necessary" refers to methods and techniques for simulating the movements of a designed avatar, verifying its suitability and functionality, and correcting any problems.

[0218] "Means of performing specific tasks" refers to methods or technologies that enable an avatar to perform specific tasks or activities on behalf of the user.

[0219] "Means for collecting execution logs and performing performance evaluations" refers to methods and technologies for collecting records of tasks performed by avatars and evaluating their results and efficiency.

[0220] "Methods for improving avatar performance based on feedback" refers to methods and technologies for improving the behavior and performance of avatars using feedback from users and systems.

[0221] "Means of collecting users' real-time actions and gaze via smart devices" refers to methods and technologies for instantly collecting users' actions and gaze via smart devices.

[0222] "Means for analyzing user preferences and gaze information" refers to methods and techniques for analyzing collected user preferences and gaze information and extracting useful patterns and trends from that data.

[0223] "Means of recommending products and services" refers to methods and technologies for presenting and recommending the most suitable products and services to users based on analyzed information.

[0224] This invention relates to a system that collects and analyzes the knowledge, thoughts, experiences, and memories of a specific individual and generates an avatar based on that information. This system is implemented using the following hardware and software.

[0225] Hardware and software to be used

[0226] Hardware: Smart glasses, webcam, server

[0227] Software: Python, OpenCV, NLP module, product recommender engine, avatar generation module

[0228] System Description

[0229] 1. User data collection phase

[0230] The server collects real-time user activity and gaze information via smart glasses. Specifically, it captures video data from the smart glasses using OpenCV and performs analysis to identify the user's gaze and interests.

[0231] 2. Data Analysis Phase

[0232] The server analyzes the collected data using NLP modules and other analytical tools. This identifies products that users are interested in and their preferences. It also performs noise reduction and misinformation cleaning to extract important data.

[0233] 3. Avatar Generation Phase

[0234] Based on the analyzed data, a user personality model and thought patterns are generated, and an avatar is created based on these. Using the avatar generation module, the avatar, which functions as a digital representation of the user, is designed in detail. This designed avatar undergoes functional verification through simulated questions based on the user's gaze and preferences.

[0235] 4. Product Recommendation Phase

[0236] The avatar provides real-time product recommendations. As the user moves around the store wearing smart glasses, the avatar uses its recommendation engine to suggest appropriate products based on their gaze. For example, if the user looks towards the electronics section, the avatar will recommend the most suitable electronic device.

[0237] 5. Feedback Phase

[0238] We receive feedback from users about the avatar's movements and responses, and improve the avatar's performance based on that feedback. We analyze user comments and activity logs and make necessary improvements.

[0239] Specific examples of processing steps

[0240] 1. The user puts on smart glasses and enters the store.

[0241] 2. The server obtains real-time video from the smart glasses and analyzes the user's gaze using OpenCV.

[0242] 3. When a user fixates their gaze on a specific product, an NLP module analyzes their preferences and interests based on that information.

[0243] 4. An avatar is generated, and a recommender engine is used to recommend products based on the analysis results.

[0244] 5. Users provide feedback on recommended products, and the server uses that information to improve the avatar's behavior.

[0245] Example of a prompt

[0246] "Please provide the following data to the NLP Processor to analyze user preferences: {user data}"

[0247] "Provide the Recommender engine with a {product list} and recommend the best products for the user."

[0248] In this way, it becomes possible to realize product recommendations that reflect the user's real-time behavior and gaze information, and to provide advanced personalized services using avatars.

[0249] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0250] Step 1:

[0251] The user wears smart glasses, and the server acquires real-time video from the smart glasses. The input is video data from the smart glasses, and the output is video frames that are processed in real time. The server uses this frame data to detect the user's gaze and analyze the gaze information.

[0252] Step 2:

[0253] The server analyzes the acquired video data using OpenCV to identify the user's gaze information. The input is real-time video frames, and the output is the user's gaze coordinate data. Based on this gaze coordinate data, the server detects objects and products that the user may be interested in.

[0254] Step 3:

[0255] The server collects basic information related to the detected products and objects. The input is a list of products detected based on gaze coordinate data, and the output is basic information related to these products (e.g., product name, category, price, etc.). The server retrieves this information and proceeds to the next analysis stage.

[0256] Step 4:

[0257] The server uses an NLP module to analyze user response data and past purchase history to identify user preferences and interests. The input is user response data and purchase history data, and the output is profile data indicating the user's preferences and interests. Based on this profile data, the server narrows down the list of product recommendation candidates.

[0258] Step 5:

[0259] The server uses a recommender engine to suggest appropriate products based on the user's profile data and gaze information. The input is profile data and gaze coordinate data, and the output is a list of recommended products. The server generates the list of recommended products, and the avatar presents it to the user.

[0260] Step 6:

[0261] The avatar presents the user with a list of recommended products and provides detailed explanations and advice. The input is the list of recommended products, and the output is a product recommendation message to the user. The server delivers this message to the user in real time through the avatar.

[0262] Step 7:

[0263] The system provides feedback on recommended products from the user. The input is user feedback data, and the output is feedback evaluation data as an analysis result. The server collects this feedback data and uses it to inform the avatar's next actions.

[0264] Step 8:

[0265] The server uses feedback evaluation data to make improvements to enhance the avatar's performance. The input is feedback evaluation data, and the output is the improved avatar behavior model. The server applies these improvements to the avatar to prepare for the next interaction.

[0266] This will allow users to receive product recommendations in real time and enjoy personalized services through avatars.

[0267] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0268] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, more natural dialogue and advanced task handling become possible. This system is implemented through the following basic phases.

[0269] 1. User data collection phase

[0270] First, the device presents the user with an initial question. For example, it might display a question such as, "Please tell me about your work experience." The user then enters their answer into the device. For example, they might enter an answer such as, "I have 5 years of experience as a marketing manager."

[0271] The server analyzes user response data in real time and extracts important keywords and phrases. For example, information such as "5 years" and "marketing manager" may be extracted. This data is stored in a structured database.

[0272] Next, the server generates additional questions based on the important information and sends them to the terminal. The emotion engine also operates simultaneously, recognizing the user's emotional state. For example, the emotion engine analyzes the user's tone of voice and facial expressions while they are answering to determine whether the user is relaxed or tense.

[0273] 2. Data Analysis and Processing Phase

[0274] The collected data is cleaned by the server to remove noise and misinformation. Next, natural language processing (NLP) techniques are used to analyze the context and meaning of the user's writing and extract important decisions and thought processes. This analysis also incorporates data from an emotion engine, taking the user's emotional state into consideration.

[0275] Based on the extracted information, the server generates a personality model and thought patterns of the user. For example, it identifies situations in which the user is likely to feel stressed and situations in which they can make calm judgments.

[0276] 3. Avatar Generation Phase

[0277] The avatar is designed based on personality models and thought patterns generated by the server. Data provided by the emotion engine is also incorporated into this process. The terminal then verifies the avatar's behavior. For example, as a simulation, it might ask the question, "Please explain the new market strategy," and verify whether the avatar responds appropriately.

[0278] In this process, the emotion engine monitors the user's emotional state based on the avatar's responses and makes necessary adjustments. For example, if the user answers a question in a relaxed tone, the avatar will be adjusted to respond in a similar tone.

[0279] 4. Avatar Usage Phase

[0280] When the user is on a business trip or busy, the terminal causes the avatar to execute specific tasks. For example, the avatar substitutes for a presentation at a meeting. The server monitors the actions and response content of the avatar in real time and collects execution logs.

[0281] The server analyzes the collected logs and evaluates the performance of the avatar. This evaluation also includes the user's emotion data provided by the emotion engine. For example, it is confirmed whether the avatar was not nervous during the presentation.

[0282] The user provides feedback on the performance of the avatar through the terminal. The server improves the performance of the avatar based on this feedback. For example, adjustments are made to maintain a more relaxed tone in the next presentation.

[0283] 5. Utilization of the Emotion Engine

[0284] The emotion engine adjusts the responses and actions of the avatar based on emotion data. For example, in a situation where the user is emotionally excited, the avatar is set to respond in a more calm tone. This makes the communication with the user more natural and effective.

[0285] As a specific example, it can be cited that when a corporate leader is on a long business trip, the avatar attends an in-house meeting instead, and makes decisions based on the leader's thinking process and judgment criteria. At this time, the emotion engine detects the tension and stress that the leader may feel during the meeting, and the avatar makes a calm response corresponding to this, so that the meeting proceeds smoothly.

[0286] As described above, the present invention provides a system that reproduces as an avatar by digitizing personal knowledge and experience and combining an emotion engine. Thereby, even when the user is absent or after death, it becomes possible to continuously utilize the knowledge and experience, and the communication becomes more natural and effective by the emotion engine.

[0287] The process flow will be described below.

[0288] Step 1:

[0289] The terminal presents an initial question to the user. For example, a question such as "Please tell me about your work experience" is displayed.

[0290] Step 2:

[0291] The user inputs an answer to the terminal. For example, the user inputs "I have 5 years of experience as a marketing manager".

[0292] Step 3:

[0293] The server analyzes the user's answer data in real time and extracts important keywords and phrases. For example, "5 years" and "marketing manager" are extracted.

[0294] Step 4:

[0295] The terminal uses an emotion engine to recognize the user's emotion in real time. For example, by analyzing the user's voice tone and expression, it determines whether the user is relaxed or tense.

[0296] Step 5:

[0297] Based on the analyzed data and the data from the emotion engine, the server generates additional questions and sends them to the terminal. For example, it asks "As a marketing manager, what projects have you led?"

[0298] Step 6:

[0299] The user inputs an answer to the additional question to the terminal. For example, the user answers "I led a marketing campaign for a new product".

[0300] Step 7:

[0301] The server also analyzes the additional response data, extracts important information, and stores it in the database. Also, a cleaning process is performed to remove noise and incorrect information.

[0302] Step 8:

[0303] The server uses natural language processing (NLP) technology to analyze the data, understand the context and meaning, and extract important decisions and thought processes. The data of the emotion engine is also incorporated into this analysis.

[0304] Step 9:

[0305] Based on the information extracted by the server, a user personality model and thought pattern are generated. The user's emotional state is taken into consideration in this process. For example, situations where the user is prone to stress or situations where the user makes decisions calmly are identified.

[0306] Step 10:

[0307] An avatar is designed based on the personality model and thought pattern generated by the server. The data of the emotion engine is also reflected in this design.

[0308] Step 11:

[0309] The terminal asks the avatar simulation questions in order to check the operation of the avatar. For example, ask a question like "Please explain the new market strategy".

[0310] Step 12:

[0311] The terminal checks the response of the avatar and verifies whether the response meets the user's expectations. The server adjusts the operation of the avatar as needed.

[0312] Step 13:

[0313] The emotion engine monitors the user's emotional state based on the avatar's responses and adjusts the avatar's movements and response tone accordingly. For example, it can be configured to have the avatar respond in a relaxed tone.

[0314] Step 14:

[0315] The device allows the user to finalize the avatar and collects any necessary changes or additional information. This ensures that the avatar behaves as the user expects.

[0316] Step 15:

[0317] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar could give a presentation at a meeting on their behalf.

[0318] Step 16:

[0319] The server monitors the avatar's movements and responses in real time and collects execution logs.

[0320] Step 17:

[0321] The server analyzes the collected logs to evaluate the avatar's performance. This evaluation also includes user emotion data provided by the emotion engine.

[0322] Step 18:

[0323] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance. For example, it might make adjustments to maintain a more relaxed tone in the next presentation.

[0324] Step 19:

[0325] The server periodically collects new information from users and updates avatar data to the latest version. This ensures that avatars always reflect the most up-to-date knowledge and experience.

[0326] Through the steps described above, the system of the present invention reproduces the user's knowledge and experience as an avatar, and by combining it with an emotion engine, it becomes possible to have natural conversations and task responses that take into account the user's emotional state.

[0327] (Example 2)

[0328] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0329] Conventional avatar systems that utilize personal information require not only effective reproduction of the user's knowledge and experience, but also natural dialogue and task execution that takes into account the user's emotional state. However, current technology is insufficient for emotion analysis, making it difficult to adjust responses and actions in response to fluctuations in the user's emotions. Furthermore, performance evaluation of tasks performed by avatars and improvements based on feedback are not effectively carried out, thus failing to improve the quality of the user experience. In addition, cleaning processes and the use of natural language processing technologies are necessary to maintain the quality of collected data.

[0330] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for asking a number of questions to collect an individual's knowledge, thoughts, experiences, and memories; means for analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its movements, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for analyzing emotional data and adjusting the avatar's responses and movements according to the user's emotional state; means for data cleaning to remove noise and misinformation; means for analyzing context and meaning using natural language processing technology and extracting important decisions and thought processes; and means for analyzing the collected emotional data using an emotion engine. This enables natural and effective dialogue and task execution that takes into account not only the user's knowledge and experience but also their emotional state.

[0331] "Personal knowledge, thoughts, experiences, and memories" refers to the specialized knowledge, thought patterns, specific experiences and events, and memories associated with them that an individual possesses.

[0332] "Means of asking questions" refers to devices and programs for presenting questions to users and collecting their answers.

[0333] "Means for analyzing response data in real time" refers to devices and programs that process responses obtained from users immediately and extract and analyze necessary information.

[0334] A "database" refers to a system for organizing and storing collected data.

[0335] "Means for generating personality models and thought patterns" refers to devices and programs that model an individual's personality and thought patterns from collected data.

[0336] An "avatar" refers to a character that acts as a representative of a user in a digital environment.

[0337] "Means for simulating and verifying actions" refers to devices and programs that virtually test the actions of an avatar and evaluate their suitability.

[0338] "Means for performing specific tasks" refers to devices and programs that allow an avatar to perform specific roles or tasks on behalf of the user.

[0339] "Means for collecting execution logs and performing performance evaluation" refers to a device and program that records the avatar's behavior history and evaluates its performance.

[0340] "Means of improving avatar performance based on feedback" refers to devices and programs that improve avatar functionality by incorporating evaluations and opinions from users.

[0341] "Means for analyzing emotional data" refers to devices and programs that recognize a user's emotional state and analyze that data.

[0342] "Data cleaning means" refers to devices and programs that remove noise and misinformation from collected data and improve data quality.

[0343] "Natural language processing technology" refers to the technology used to understand, process, and generate human language using computers.

[0344] "Means for analyzing collected emotional data" refers to devices and programs that process emotional data obtained from users and understand its content.

[0345] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and has it function as a proxy for the user. Furthermore, it integrates an emotion engine that recognizes the user's emotions, enabling more natural dialogue and advanced task handling. This system is implemented through the following basic phases.

[0346] User data collection phase

[0347] The terminal presents the user with an initial question such as, "Please tell me about your work experience." The user enters a response into the terminal, such as, "I have 5 years of experience as a marketing manager." The server analyzes this response data in real time, extracting important keywords and phrases. This data is stored in a structured database. The server then generates additional questions and sends them to the terminal. An emotion engine also operates, analyzing the user's emotional state. For example, it analyzes the user's tone of voice and facial expressions while they are answering to determine whether they are relaxed or nervous.

[0348] Data Analysis and Processing Phase

[0349] The server cleans the collected data, removing noise and misinformation. Using natural language processing (NLP) techniques, it analyzes the context and meaning of the user's writing to extract important decisions and thought processes. This analysis also incorporates data from the emotion engine, taking the user's emotional state into account. Based on the extracted information, the server generates a personality model and thought patterns of the user.

[0350] Avatar generation phase

[0351] The avatar is designed based on personality models and thought patterns generated by the server. Data provided by the emotion engine is also reflected. The terminal verifies the avatar's behavior. For example, it performs a simulation asking, "Please explain the new market strategy," and verifies whether the avatar responds appropriately. During this process, the emotion engine monitors the user's emotional state based on the avatar's responses and makes necessary adjustments.

[0352] Avatar Usage Phase

[0353] When a user is away on a business trip or busy, their device can have an avatar perform specific tasks. For example, the avatar can give a presentation at a meeting. A server monitors the avatar's actions and responses in real time and collects execution logs. The server analyzes the collected logs and evaluates the avatar's performance. This evaluation also includes user emotion data provided by an emotion engine. For example, it checks whether the avatar was nervous during the presentation. The user provides feedback through their device, and the server uses that feedback to improve the avatar's performance.

[0354] Utilizing the Emotion Engine

[0355] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar will respond in a calmer tone. As a concrete example, when a company leader is on a long business trip, the avatar attends internal meetings on their behalf and makes decisions based on the leader's thought process and judgment criteria. In this scenario, the emotion engine detects any tension or stress the leader might be experiencing, and the avatar responds accordingly with a calmer tone.

[0356] Specific example

[0357] The following scenario is an example of a prompt message.

[0358] "This scenario illustrates a situation where a company leader is on a long-term business trip, but their avatar attends an internal meeting and makes decisions based on the leader's thinking and decision-making criteria. During the meeting, an emotion engine analyzes the leader's emotional data to detect tension and stress. Based on this, the avatar provides calm responses, ensuring the meeting runs smoothly."

[0359] Hardware and software to be used

[0360] Hardware: Devices (PCs, smartphones, tablets, etc.), servers

[0361] Software: Natural Language Processing (NLP) engine, emotion engine, database system

[0362] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0363] Step 1:

[0364] Presentation of initial questions

[0365] The device displays an initial question such as, "Please tell us about your work experience." The user enters their answer using the keyboard or touch input. Specifically, a text box appears on the device screen, and the user enters, "I have 5 years of experience as a marketing manager." This input is saved on the device as text data.

[0366] Step 2:

[0367] Collection of user responses

[0368] The device collects user responses and sends them to the server. For example, it might transfer the user's response data, "I have 5 years of experience as a marketing manager," to the server. This transmitted data becomes input data for the server's analysis system.

[0369] Step 3:

[0370] Analysis of response data and keyword extraction

[0371] The server analyzes user response data in real time and extracts important keywords and phrases. Specifically, it uses natural language processing (NLP) algorithms to extract keywords such as "5 years" and "marketing manager." The input data is the user's response text, and the extracted keywords become the output data. This extracted data is structured and stored in a database.

[0372] Step 4:

[0373] Generating and submitting additional questions

[0374] The server generates additional questions based on the extracted keywords and sends them to the terminal. For example, an additional question such as "Please tell us about any specific projects or successful campaigns you have worked on recently" might be generated. The input data is the extracted keywords, and the generated additional questions become the output data.

[0375] Step 5:

[0376] Emotional analysis

[0377] The emotion engine analyzes the user's emotional state. Specifically, it analyzes voice tone and facial expression data to determine whether the user is relaxed or tense. The input data consists of recordings of the user's voice and facial expressions, and the output data is the emotional state (e.g., relaxed, tense).

[0378] Step 6:

[0379] Data Cleaning

[0380] The server cleans the collected data, removing noise and misinformation. For example, it filters out grammatical errors and irrelevant information from the input data. The input data is the user's text responses, and the cleaned text data becomes the output data.

[0381] Step 7:

[0382] Contextual analysis using natural language processing

[0383] The server uses an NLP engine to analyze the context and meaning of the user's text. Specifically, it analyzes text data to extract contextual information such as "the user has had successful marketing experiences." The input data is cleaned text data, and the contextual information becomes the output data.

[0384] Step 8:

[0385] Extraction of important decisions and thought processes

[0386] The server extracts important user decisions and thought processes from the analysis results. For example, it identifies situations where the user feels stressed or situations where they make calm judgments. The input data is the analyzed contextual information, and the characteristics of the thought process become the output data.

[0387] Step 9:

[0388] Generating personality models and thought patterns

[0389] The server generates personality models and thought patterns based on extracted thought processes. For example, it might create a model such as "Users tend to make calm judgments." The input data is the characteristics of the thought process, and the generated personality model becomes the output data.

[0390] Step 10:

[0391] Avatar design

[0392] The server designs the avatar based on the generated personality model and thought patterns. It also incorporates data from the emotion engine. The input data consists of the personality model and emotion data, while the avatar specifications become the output data.

[0393] Step 11:

[0394] Avatar operation check

[0395] The device performs a simulation, such as asking the avatar to "explain the new market strategy," to verify its operation. For example, the device asks a question, and the system checks whether the avatar answers appropriately. The input data is the question used for the simulation, and the response results become the output data.

[0396] Step 12:

[0397] Executing a specific task

[0398] The device instructs an avatar to perform specific tasks, such as giving a presentation. For example, "The avatar explains the marketing strategy for a new product at a meeting." The input data is the task content, and the output data is the result of the execution.

[0399] Step 13:

[0400] Real-time monitoring and log collection

[0401] The server monitors the avatar's actions and responses in real time and collects execution logs. The input data consists of the avatar's actions and responses, while the execution logs are the output data.

[0402] Step 14:

[0403] Performance evaluation and feedback

[0404] The server analyzes collected logs to evaluate avatar performance. It also considers user feedback to improve avatar performance. Input data consists of execution logs and feedback, while the output data represents the improvements made.

[0405] Step 15:

[0406] Adjustment based on emotional data

[0407] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar is set to respond in a calm tone. The input data is emotional data, and the adjusted responses become the output data.

[0408] (Application Example 2)

[0409] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0410] Modern virtual stores require real-time, personalized service to meet user needs for product recommendations and interactive experiences. However, traditional systems struggle to analyze users' emotional states and respond appropriately, sometimes leading to decreased satisfaction. They also struggle to provide personalized product recommendations based on individual user needs. Therefore, there is a need for a system that improves the quality of user interaction and allows users to have an experience in virtual stores that closely resembles that of physical stores.

[0411] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0412] In this invention, the server includes means for asking multiple questions to collect an individual's knowledge, thoughts, experiences, and memories; means for collecting and analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance, and for improving the avatar's performance based on feedback; means for interacting with the user using a smartphone and making product suggestions in a virtual store; and means for analyzing the user's emotional state using an emotion engine and adjusting the avatar's responses. This makes it possible to provide highly personalized product suggestions and services that take into account the user's emotional state.

[0413] "Personal knowledge" refers to the specialized information and experience that individual people possess.

[0414] "Thinking" refers to the process of problem-solving and decision-making that users engage in.

[0415] "Experience" refers to the knowledge and skills that an individual has learned from their past actions and experiences.

[0416] "Memory" refers to an individual's ability to retain past events and information.

[0417] A "question" refers to a question posed to a user in order to obtain information.

[0418] "Response data" refers to the information and responses that users provide in response to questions.

[0419] "Real-time analysis" refers to the process of analyzing data immediately on the spot.

[0420] "Important information" refers to data that is deemed particularly valuable among the analyzed data.

[0421] A "structured database" refers to a database that is organized and stored according to a specific format or set of rules.

[0422] A "personality model" refers to a model that represents the personality traits of individual users.

[0423] "Thinking patterns" refer to the series of actions and decision-making tendencies that users exhibit when solving problems.

[0424] An "avatar" refers to a virtual person or character that acts on behalf of a user.

[0425] "Simulation" refers to the act of virtually recreating a real environment to verify its operation.

[0426] "Performing a specific task" refers to carrying out pre-defined tasks or duties.

[0427] An "execution log" refers to a record of the actions and responses performed by the avatar.

[0428] "Performance evaluation" refers to the process of evaluating the performance of an avatar.

[0429] A "smartphone" refers to a portable, multi-functional information terminal.

[0430] "Dialogue" refers to communication between the user and the system.

[0431] A "virtual store" refers to a virtual sales environment that exists on the internet.

[0432] "Product recommendation" refers to the act of recommending appropriate products to users.

[0433] An "emotion engine" refers to a technology that analyzes a user's emotional state and adjusts its response accordingly.

[0434] "Adjusting your response" means changing your answer to suit the other person's situation and feelings.

[0435] This invention relates to a system that provides personalized product suggestions to users in a virtual store and adjusts responses according to their emotions. Specifically, it provides a system that collects and analyzes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and has the avatar perform specific tasks on behalf of the user. The specific configuration and method for carrying out this invention are described below.

[0436] Hardware and software

[0437] This system consists of a smartphone, a server, a natural language processing engine (NLP), an emotion engine, a database, and a front-end application.

[0438] Hardware: Smartphone (iOS or Android® device)

[0439] software:

[0440] Natural Language Processing Engine (NLP): Uses the Google NLP API.

[0441] Emotion engine: Uses Microsoft® Azure® Emotion API.

[0442] Database: Firebase Realtime Database is used.

[0443] Frontend application: Developed using React Native.

[0444] Data collection and analysis

[0445] In the initial stages of the system, the server collects data about the user's knowledge, thoughts, experiences, and memories. This involves a process where the user is asked multiple questions via their smartphone and provides answers. This response data is analyzed in real time by the server, and important information is extracted.

[0446] Data storage and modeling

[0447] The analyzed data is filtered to remove noise and misinformation before being stored in a structured format in the Firebase Realtime Database. Then, natural language processing (NLP) techniques are used to analyze the context and meaning, generating a user personality model and thought patterns.

[0448] Avatar generation and simulation

[0449] The avatar is designed based on personality models and thought patterns generated by the server. The avatar design also incorporates data from an emotion engine to reflect the user's emotional state. Finally, the avatar's movements are simulated and verified.

[0450] Using avatars

[0451] When a user uses a virtual store, an avatar performs specific tasks on their behalf, such as suggesting products. Specifically, it interacts with the user using a smartphone, analyzes the user's emotional state using an emotion engine, and adjusts its response accordingly.

[0452] Feedback and Performance Evaluation

[0453] Avatar execution logs are collected and performance is evaluated. This includes user feedback, and the collected feedback can be used to improve the avatar's performance. This feedback loop allows for continuous improvement of the avatar's behavior and responsiveness.

[0454] Examples of specific cases and prompt statements

[0455] The following interactions are examples of what can be considered.

[0456] Example of a prompt:

[0457] "Please tell me about your work experience."

[0458] User response:

[0459] "I have five years of experience as a marketing manager."

[0460] The data collected in this way is analyzed, and an avatar is generated that interacts with the user in the virtual store based on the user's personality model. This makes it possible to provide highly personalized product suggestions and services that take into account the user's emotional state.

[0461] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0462] Step 1:

[0463] The server initiates a process to collect the user's knowledge, thoughts, experiences, and memories. Specifically, it asks the user multiple questions via their device (smartphone). The user's responses to these questions are collected and analyzed in real time.

[0464] Input: Response data entered by the user on the device.

[0465] Output: Set of collected response data

[0466] Step 2:

[0467] The server analyzes the collected response data in real time and extracts important information. It uses the Google NLP API to analyze the context and meaning of the response data and extract key keywords and phrases.

[0468] Input: Collected response data

[0469] Output: Important keywords and phrases

[0470] Step 3:

[0471] The server stores the extracted key keywords and phrases in a structured database (Firebase Realtime Database). This data will be used for future analysis and avatar generation.

[0472] Input: Important keywords or phrases

[0473] Output: Structured data stored in the database

[0474] Step 4:

[0475] The server generates individual personality models and thought patterns based on stored data. Natural language processing (NLP) techniques are used to model user characteristics and patterns.

[0476] Input: Structured data stored in a database

[0477] Output: Personality model and thought patterns

[0478] Step 5:

[0479] The server designs avatars based on generated personality models and thought patterns. It also integrates data from emotion engines such as the Microsoft Azure Emotion API to consider emotion-based responses.

[0480] Input: Data from personality models, thought patterns, and emotion engines.

[0481] Output: Designed avatar

[0482] Step 6:

[0483] The server simulates and verifies the avatar's movements. This process verifies whether the avatar can respond appropriately and perform specific tasks. Adjustments are made as needed.

[0484] Input: Designed avatar

[0485] Output: Avatars whose operation has been confirmed

[0486] Step 7:

[0487] When a user uses a virtual store, an avatar performs specific tasks on their behalf, such as suggesting products, via their device. The avatar interacts with the user using a smartphone, analyzes the user's emotional state using an emotion engine, and adjusts its responses accordingly.

[0488] Input: User questions, requests, and sentiment data

[0489] Output: Product suggestions and responses

[0490] Step 8:

[0491] The server collects avatar execution logs and performs performance evaluations. Based on the collected data, including user feedback, the avatar's performance is improved.

[0492] Input: Avatar execution log, user feedback

[0493] Output: Evaluation results, improved avatar performance

[0494] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0495] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0496] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0497] [Second Embodiment]

[0498] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0499] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0500] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0501] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0502] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0503] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0504] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0505] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0506] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0507] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0508] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0509] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0510] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. This system is implemented through the following basic phases.

[0511] 1. User data collection phase

[0512] First, the terminal asks the user an initial question. For example, it might ask, "Please tell me about your work experience." The user then enters their answer into the terminal. For example, they might answer, "I have 5 years of experience as a marketing manager." The server then analyzes the user's response data in real time and extracts important keywords and phrases. This analyzed data is immediately stored in a structured database.

[0513] Next, the server generates additional questions based on important information from the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?" The user's additional responses are also saved.

[0514] 2. Data Analysis and Processing Phase

[0515] The collected data is first cleaned by the server to remove noise and misinformation. Then, using natural language processing (NLP) techniques, the context and meaning of the user's writing are analyzed, and important decisions and thought processes are extracted. Based on this information, the server generates a personality model and thought patterns of the user. This allows for a systematic understanding of what kind of decisions the user makes in what situations.

[0516] 3. Avatar Generation Phase

[0517] The server designs an avatar based on a personality model and thought patterns it generates. This designed avatar actively utilizes the user's past data and functions as a representation of the user. Next, the terminal verifies the avatar's operation. For example, it might ask a simulated question such as, "Please explain the new market strategy." The terminal checks whether the avatar's response meets the user's expectations, and the server makes adjustments if necessary.

[0518] 4. Avatar Usage Phase

[0519] When a user is away on a business trip or too busy, the avatar can perform specific tasks on their behalf. For example, the avatar can give a presentation at a meeting as a task set by the user. The device monitors the avatar's actions and responses in real time and sends the execution logs to the server. The server analyzes these logs and evaluates the avatar's performance. Based on this evaluation, the server collects feedback from the user and improves the avatar's performance.

[0520] This allows the user's knowledge and experience to be continuously utilized even when they are absent or after their death. Furthermore, the avatar's performance can be continuously improved based on feedback, always reflecting the latest knowledge and experience.

[0521] As a concrete example, when a company leader is on a long business trip, an avatar can attend internal meetings on their behalf and make decisions based on the leader's thought process and decision-making criteria. This allows the company to maximize the use of the leader's expertise and experience even in their absence.

[0522] As described above, the present invention provides a system that can be widely applied by digitizing an individual's knowledge and experience and reproducing it as an avatar.

[0523] The following describes the processing flow.

[0524] Step 1:

[0525] The device presents the user with an initial question. For example, it might display a question such as, "Please tell us about your work experience."

[0526] Step 2:

[0527] The user enters their response into the device. For example, they might enter, "I have 5 years of experience as a marketing manager."

[0528] Step 3:

[0529] The server analyzes user response data in real time and extracts important keywords and phrases. For example, it might extract "5 years" or "marketing manager."

[0530] Step 4:

[0531] The server generates additional questions based on the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?"

[0532] Step 5:

[0533] The user enters their answer to an additional question on the device. For example, they might answer, "I led the marketing campaign for a new product."

[0534] Step 6:

[0535] The server also analyzes the additional response data, extracts important information, and saves it to the database. It also generates additional questions as needed.

[0536] Step 7:

[0537] The server cleans the collected data, removing noise and misinformation to maintain data quality.

[0538] Step 8:

[0539] The server analyzes the cleaned data using natural language processing (NLP) techniques to understand the context and meaning, and extract important decisions and thought processes.

[0540] Step 9:

[0541] The server generates a user personality model and thought patterns based on the extracted information. This allows for a systematic understanding of how users make decisions in different situations.

[0542] Step 10:

[0543] The avatar is designed based on personality models and thought patterns generated by the server. This process incorporates the user's past data.

[0544] Step 11:

[0545] The device asks the avatar simulation questions to test its functionality. For example, it might ask, "Please explain your new market strategy."

[0546] Step 12:

[0547] The terminal checks the avatar's response and verifies that it is operating according to the specifications. The server makes adjustments as needed.

[0548] Step 13:

[0549] The device performs a final confirmation with the user to verify that there are no problems with the avatar's operation. This ensures that the avatar operates according to the user's expectations.

[0550] Step 14:

[0551] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar could give a presentation at a meeting on their behalf.

[0552] Step 15:

[0553] The server collects avatar execution logs, recording the avatar's actions and responses.

[0554] Step 16:

[0555] The system analyzes the logs collected by the server to evaluate the avatar's performance. This verifies whether the avatar was able to perform the task properly.

[0556] Step 17:

[0557] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance.

[0558] Step 18:

[0559] The server periodically collects new information from users and updates avatar data to the latest version. This ensures that avatars always reflect the most up-to-date knowledge and experience.

[0560] (Example 1)

[0561] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0562] Currently, many companies and individuals are finding it difficult to digitize their knowledge and experience and utilize that data effectively. In particular, problems arise when individuals are unavailable or too busy to provide support based on their experience and knowledge, especially when making important decisions or judgments. Furthermore, removing noise and misinformation from collected data, and improving the accuracy of data analysis, remain challenges.

[0563] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0564] In this invention, the server includes means for presenting information for asking multiple questions; means for collecting and analyzing an individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for using the avatar to perform specific tasks on behalf of the individual; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for cleaning the collected data from noise and misinformation; and means for analyzing the context and meaning of the cleaned data using natural language processing technology and extracting important decisions and thought processes. This makes it possible to effectively digitize an individual's knowledge and experience and support important decision-making and judgment through the avatar. Furthermore, it allows the individual's knowledge and experience to continue to be utilized even when they are absent or busy.

[0565] An "information presentation means" is a device that presents questions or instructions to the user and collects answers or information.

[0566] "Response data" refers to the information or content that users provide or input to information presentation tools.

[0567] "Methods for real-time analysis" refers to the process and technology of immediately analyzing the content of response data as soon as it is obtained and extracting the necessary information.

[0568] A "structured database" refers to a database where the stored data is organized in a specific format so that it can be easily searched and analyzed.

[0569] A "personality model" is a data-based representation of an individual's personality and behavioral characteristics.

[0570] A "thinking pattern" is a model that extracts and represents the characteristics of an individual's judgment criteria and decision-making process.

[0571] A "simulation method" is a means of testing in advance how an avatar will actually behave.

[0572] A "proxy means" is a method by which an avatar performs a specific task on behalf of an individual.

[0573] An "execution log" refers to a detailed record of when an avatar performs a task.

[0574] "Performance evaluation" is a process that evaluates the quality of the avatar's movements and responses based on execution logs.

[0575] "Feedback" refers to evaluations and opinions regarding the results of task execution or the performance of the avatar.

[0576] "Cleaning methods" refer to techniques and processes that remove noise and misinformation from collected data, thereby improving data quality.

[0577] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human natural language.

[0578] "Important decisions and thought processes" refer to the key information and processes involved in making a particular idea or judgment.

[0579] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. This system is implemented through the following basic phases.

[0580] 1. User data collection phase

[0581] First, the terminal asks the user an initial question. For example, it might ask, "Please tell me about your work experience." The user then enters their answer into the terminal. For example, they might answer, "I have five years of experience as a marketing manager." The server then analyzes the user's response data in real time and extracts important keywords and phrases. This analyzed data is immediately stored in a structured database. Next, the server generates additional questions based on the important information from the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects did you lead?" The user's additional response data is also stored in the same way.

[0582] 2. Data Analysis and Processing Phase

[0583] The collected data is first cleaned by the server to remove noise and misinformation. Then, using natural language processing (NLP) techniques, the context and meaning of the user's writing are analyzed, and important decisions and thought processes are extracted. Based on this information, the server generates a personality model and thought patterns of the user. This allows for a systematic understanding of what kind of decisions the user makes in what situations.

[0584] 3. Avatar Generation Phase

[0585] The server designs an avatar based on a personality model and thought patterns it generates. This designed avatar actively utilizes the user's past data and functions as a representation of the user. Next, the terminal verifies the avatar's operation. For example, it might ask a simulated question such as, "Please explain the new market strategy." The terminal checks whether the avatar's response meets the user's expectations, and the server makes adjustments if necessary.

[0586] 4. Avatar Usage Phase

[0587] When a user is away on a business trip or busy, the avatar performs specific tasks on their behalf. For example, the avatar can give a presentation at a meeting as a task set by the user. The terminal monitors the avatar's actions and responses in real time and sends the execution log to the server. The server analyzes this log and evaluates the avatar's performance. Based on this evaluation, feedback from the user is collected, and the server improves the avatar's performance. This makes it possible to continuously utilize the user's knowledge and experience even when the user is absent or after their death. Furthermore, the avatar's performance continuously improves based on feedback, always reflecting the latest knowledge and experience.

[0588] Hardware and software to be used

[0589] Devices: PC, smartphone, tablet

[0590] Servers: High-performance server clusters (e.g., AWS EC2, Google Cloud Platform)

[0591] Software technologies: Natural language processing (e.g., Spacy, BERT), database management systems (e.g., MySQL, PostgreSQL)

[0592] Examples of specific cases and prompt statements

[0593] As a concrete example, consider a scenario where a company leader is on a long business trip, and an avatar attends internal meetings on their behalf, making decisions based on the leader's thought process and decision-making criteria. In this case, the leader can assign the following tasks to the avatar from their business trip location.

[0594] Examples of prompts to input into a generative AI model

[0595] Now, please give a presentation on our new market strategy at today's meeting. Please base your presentation on the following points:

[0596] 1. The importance and potential benefits of new markets

[0597] 2. Our strengths based on competitive analysis

[0598] 3. Specific action plan and timeline

[0599] In response to this prompt, the avatar can deliver a natural presentation based on the leader's past data and thought patterns generated by the system. The terminal monitors the avatar's actions in real time and sends feedback to the server to improve the avatar's performance.

[0600] With the above configuration, the present invention provides a system that can be widely applied by digitizing an individual's knowledge and experience and reproducing it as an avatar.

[0601] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0602] Step 1: User Data Collection Phase

[0603] Specific actions:

[0604] The device asks the user initial questions. For example, it might ask, "Please tell me about your work experience."

[0605] Input: User's response (e.g., "I have 5 years of experience as a marketing manager")

[0606] Data processing: The server analyzes this response in real time and extracts important keywords and phrases.

[0607] Output: Analyzed data (e.g., "5 years", "Marketing Manager")

[0608] Specific actions:

[0609] The server saves the analysis data to the database.

[0610] Input: Analyzed data

[0611] Output: Structured data stored in the database

[0612] Specific actions:

[0613] The server generates additional questions based on the analyzed data and sends them to the terminal.

[0614] Input: Analysis data

[0615] Data processing: Generating additional questions (e.g., "As a marketing manager, what projects did you lead?")

[0616] Output: Additional questions

[0617] Step 2: Data Analysis and Processing Phase

[0618] Specific actions:

[0619] The server cleans the collected data, removing noise and misinformation.

[0620] Input: Collected data

[0621] Data processing: Data cleaning

[0622] Output: Cleaned data

[0623] Specific actions:

[0624] The server uses NLP (Neuro-Linguistic Programming) technology to analyze context and meaning.

[0625] Input: Cleaned data

[0626] Data processing: Contextual analysis through natural language processing (e.g., using the BERT model)

[0627] Output: Information with context and meaning analyzed.

[0628] Specific actions:

[0629] The server generates a user personality model and thought patterns from the analysis results.

[0630] Input: Information analyzed for context and meaning

[0631] Data processing: Generation of personality models and thought patterns

[0632] Output: User personality model and thought patterns

[0633] Step 3: Avatar Generation Phase

[0634] Specific actions:

[0635] The avatar is designed based on personality models and thought patterns generated by the server.

[0636] Input: User's personality model and thought patterns

[0637] Data processing: Avatar design

[0638] Output: Designed avatar

[0639] Specific actions:

[0640] The device will perform a function check on the avatar.

[0641] Input: Designed avatar

[0642] Data calculation: Ask simulation questions and observe the avatar's responses (e.g., "Please explain the new market strategy").

[0643] Output: Avatar response data

[0644] Specific actions:

[0645] The server will adjust the avatar as needed.

[0646] Input: Avatar response data

[0647] Data processing: Confirmation and adjustment of responses

[0648] Output: Adjusted avatar

[0649] Step 4: Avatar Utilization Phase

[0650] Specific actions:

[0651] The avatar performs tasks set by the user.

[0652] Input: User's task settings (e.g., "Presentation at a meeting")

[0653] Data processing: Avatar performs tasks

[0654] Output: Task execution results

[0655] Specific actions:

[0656] The device monitors the avatar's movements and responses in real time and collects logs.

[0657] Input: Avatar movement data

[0658] Data processing: Collection of operation logs

[0659] Output: Collected execution logs

[0660] Specific actions:

[0661] The server analyzes the execution logs and performs a performance evaluation.

[0662] Input: Collected execution logs

[0663] Data processing: Performance evaluation

[0664] Output: Evaluation results and necessary improvements

[0665] Specific actions:

[0666] The server improves avatar performance based on user feedback.

[0667] Input: User feedback and evaluation results

[0668] Data processing: Improving avatar performance based on feedback.

[0669] Output: Improved performance avatar

[0670] (Application Example 1)

[0671] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0672] In recent years, there has been a growing demand for technology that digitizes an individual's knowledge and experience, and uses that data to generate avatars that function as digital representations. However, current technology lacks an effective means of identifying in real time what products and services a user is interested in and recommending appropriate products and services based on those preferences. Furthermore, there is a lack of means to quickly improve the avatar's behavior and responses based on user feedback. Solving these challenges is essential.

[0673] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0674] In this invention, the server includes means for asking multiple questions to collect an individual's knowledge, thoughts, experiences, and memories; means for collecting and analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for collecting the user's real-time movements and gaze via a smart terminal and analyzing the user's preferences and gaze information; and means for generating the user's avatar based on the analyzed information and recommending products and services in real time. This makes it possible to recommend products and services that reflect the user's preferences and gaze in real time.

[0675] "Personal knowledge, thoughts, experiences, and memories" refers to the collection of information, ways of thinking, actual activities and processes performed, and past experiences of a particular individual.

[0676] "Means of asking questions" refers to methods and techniques for presenting a variety of questions in order to obtain information from an individual.

[0677] "Means for collecting and analyzing response data in real time" refers to methods and technologies for quickly collecting individual responses to questions and immediately analyzing that data.

[0678] "Means for extracting important information from analyzed data and storing it in a structured database" refers to methods and techniques for analyzing collected data, identifying key elements, and storing them in a systematic database.

[0679] "Methods for generating personality models and thought patterns" refer to methods and techniques for modeling an individual's personality and thought patterns based on their behavior and responses.

[0680] "Methods for designing avatars" refers to methods and technologies for designing and creating avatars that function as extensions of an individual, based on generated personality models and thought patterns.

[0681] "Means of performing operational checks and making adjustments as necessary" refers to methods and techniques for simulating the movements of a designed avatar, verifying its suitability and functionality, and correcting any problems.

[0682] "Means of performing specific tasks" refers to methods or technologies that enable an avatar to perform specific tasks or activities on behalf of the user.

[0683] "Means for collecting execution logs and performing performance evaluations" refers to methods and technologies for collecting records of tasks performed by avatars and evaluating their results and efficiency.

[0684] "Methods for improving avatar performance based on feedback" refers to methods and technologies for improving the behavior and performance of avatars using feedback from users and systems.

[0685] "Means of collecting users' real-time actions and gaze via smart devices" refers to methods and technologies for instantly collecting users' actions and gaze via smart devices.

[0686] "Means for analyzing user preferences and gaze information" refers to methods and techniques for analyzing collected user preferences and gaze information and extracting useful patterns and trends from that data.

[0687] "Means of recommending products and services" refers to methods and technologies for presenting and recommending the most suitable products and services to users based on analyzed information.

[0688] This invention relates to a system that collects and analyzes the knowledge, thoughts, experiences, and memories of a specific individual and generates an avatar based on that information. This system is implemented using the following hardware and software.

[0689] Hardware and software to be used

[0690] Hardware: Smart glasses, webcam, server

[0691] Software: Python, OpenCV, NLP module, product recommender engine, avatar generation module

[0692] System Description

[0693] 1. User data collection phase

[0694] The server collects real-time user activity and gaze information via smart glasses. Specifically, it captures video data from the smart glasses using OpenCV and performs analysis to identify the user's gaze and interests.

[0695] 2. Data Analysis Phase

[0696] The server analyzes the collected data using NLP modules and other analytical tools. This identifies products that users are interested in and their preferences. It also performs noise reduction and misinformation cleaning to extract important data.

[0697] 3. Avatar Generation Phase

[0698] Based on the analyzed data, a user personality model and thought patterns are generated, and an avatar is created based on these. Using the avatar generation module, the avatar, which functions as a digital representation of the user, is designed in detail. This designed avatar undergoes functional verification through simulated questions based on the user's gaze and preferences.

[0699] 4. Product Recommendation Phase

[0700] The avatar provides real-time product recommendations. As the user moves around the store wearing smart glasses, the avatar uses its recommendation engine to suggest appropriate products based on their gaze. For example, if the user looks towards the electronics section, the avatar will recommend the most suitable electronic device.

[0701] 5. Feedback Phase

[0702] We receive feedback from users about the avatar's movements and responses, and improve the avatar's performance based on that feedback. We analyze user comments and activity logs and make necessary improvements.

[0703] Specific examples of processing steps

[0704] 1. The user puts on smart glasses and enters the store.

[0705] 2. The server obtains real-time video from the smart glasses and analyzes the user's gaze using OpenCV.

[0706] 3. When a user fixates their gaze on a specific product, an NLP module analyzes their preferences and interests based on that information.

[0707] 4. An avatar is generated, and a recommender engine is used to recommend products based on the analysis results.

[0708] 5. Users provide feedback on recommended products, and the server uses that information to improve the avatar's behavior.

[0709] Example of a prompt

[0710] "Please provide the following data to the NLP Processor to analyze user preferences: {user data}"

[0711] "Provide the Recommender engine with a {product list} and recommend the best products for the user."

[0712] In this way, it becomes possible to realize product recommendations that reflect the user's real-time behavior and gaze information, and to provide advanced personalized services using avatars.

[0713] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0714] Step 1:

[0715] The user wears smart glasses, and the server acquires real-time video from the smart glasses. The input is video data from the smart glasses, and the output is video frames that are processed in real time. The server uses this frame data to detect the user's gaze and analyze the gaze information.

[0716] Step 2:

[0717] The server analyzes the acquired video data using OpenCV to identify the user's gaze information. The input is real-time video frames, and the output is the user's gaze coordinate data. Based on this gaze coordinate data, the server detects objects and products that the user may be interested in.

[0718] Step 3:

[0719] The server collects basic information related to the detected products and objects. The input is a list of products detected based on gaze coordinate data, and the output is basic information related to these products (e.g., product name, category, price, etc.). The server retrieves this information and proceeds to the next analysis stage.

[0720] Step 4:

[0721] The server uses an NLP module to analyze user response data and past purchase history to identify user preferences and interests. The input is user response data and purchase history data, and the output is profile data indicating the user's preferences and interests. Based on this profile data, the server narrows down the list of product recommendation candidates.

[0722] Step 5:

[0723] The server uses a recommender engine to suggest appropriate products based on the user's profile data and gaze information. The input is profile data and gaze coordinate data, and the output is a list of recommended products. The server generates the list of recommended products, and the avatar presents it to the user.

[0724] Step 6:

[0725] The avatar presents the user with a list of recommended products and provides detailed explanations and advice. The input is the list of recommended products, and the output is a product recommendation message to the user. The server delivers this message to the user in real time through the avatar.

[0726] Step 7:

[0727] The system provides feedback on recommended products from the user. The input is user feedback data, and the output is feedback evaluation data as an analysis result. The server collects this feedback data and uses it to inform the avatar's next actions.

[0728] Step 8:

[0729] The server uses feedback evaluation data to make improvements to enhance the avatar's performance. The input is feedback evaluation data, and the output is the improved avatar behavior model. The server applies these improvements to the avatar to prepare for the next interaction.

[0730] This will allow users to receive product recommendations in real time and enjoy personalized services through avatars.

[0731] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0732] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, more natural dialogue and advanced task handling become possible. This system is implemented through the following basic phases.

[0733] 1. User data collection phase

[0734] First, the device presents the user with an initial question. For example, it might display a question such as, "Please tell me about your work experience." The user then enters their answer into the device. For example, they might enter an answer such as, "I have 5 years of experience as a marketing manager."

[0735] The server analyzes user response data in real time and extracts important keywords and phrases. For example, information such as "5 years" and "marketing manager" may be extracted. This data is stored in a structured database.

[0736] Next, the server generates additional questions based on the important information and sends them to the terminal. The emotion engine also operates simultaneously, recognizing the user's emotional state. For example, the emotion engine analyzes the user's tone of voice and facial expressions while they are answering to determine whether the user is relaxed or tense.

[0737] 2. Data Analysis and Processing Phase

[0738] The collected data is cleaned by the server to remove noise and misinformation. Next, natural language processing (NLP) techniques are used to analyze the context and meaning of the user's writing and extract important decisions and thought processes. This analysis also incorporates data from an emotion engine, taking the user's emotional state into consideration.

[0739] Based on the extracted information, the server generates a personality model and thought patterns of the user. For example, it identifies situations in which the user is likely to feel stressed and situations in which they can make calm judgments.

[0740] 3. Avatar Generation Phase

[0741] The avatar is designed based on personality models and thought patterns generated by the server. Data provided by the emotion engine is also incorporated into this process. The terminal then verifies the avatar's behavior. For example, as a simulation, it might ask the question, "Please explain the new market strategy," and verify whether the avatar responds appropriately.

[0742] In this process, the emotion engine monitors the user's emotional state based on the avatar's responses and makes necessary adjustments. For example, if the user answers a question in a relaxed tone, the avatar will be adjusted to respond in a similar tone.

[0743] 4. Avatar Usage Phase

[0744] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar can give a presentation at a meeting. The server monitors the avatar's actions and responses in real time and collects execution logs.

[0745] The server analyzes the collected logs to evaluate the avatar's performance. This evaluation also includes user emotion data provided by the emotion engine. For example, it checks whether the avatar was nervous during the presentation.

[0746] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance. For example, it might make adjustments to maintain a more relaxed tone in the next presentation.

[0747] 5. Utilizing the Emotional Engine

[0748] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar is set to respond in a calmer tone. This makes communication with the user more natural and effective.

[0749] As a concrete example, when a company leader is on a long business trip, an avatar can attend internal meetings on their behalf and make decisions based on the leader's thought process and decision-making criteria. In this scenario, an emotional engine detects any tension or stress the leader might feel during the meeting, and the avatar responds calmly to address these issues, ensuring the meeting proceeds smoothly.

[0750] As described above, the present invention provides a system that digitizes an individual's knowledge and experience and combines it with an emotion engine to reproduce it as an avatar. This makes it possible to continuously utilize the user's knowledge and experience even when the user is absent or after death, and the emotion engine makes communication more natural and effective.

[0751] The following describes the processing flow.

[0752] Step 1:

[0753] The device presents the user with an initial question. For example, it might display a question such as, "Please tell us about your work experience."

[0754] Step 2:

[0755] The user enters their answer into the device. For example, they might enter, "I have 5 years of experience as a marketing manager."

[0756] Step 3:

[0757] The server analyzes user response data in real time and extracts important keywords and phrases. For example, it might extract "5 years" or "marketing manager."

[0758] Step 4:

[0759] The device uses an emotion engine to recognize the user's emotions in real time. For example, it analyzes the user's voice tone and facial expressions to determine whether the user is relaxed or stressed.

[0760] Step 5:

[0761] The server generates additional questions based on the analyzed data and the sentiment engine data, and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?"

[0762] Step 6:

[0763] The user enters their answer to an additional question on the device. For example, they might answer, "I led the marketing campaign for a new product."

[0764] Step 7:

[0765] The server also analyzes the additional response data, extracts important information, and saves it to the database. It also performs cleaning processes to remove noise and misinformation.

[0766] Step 8:

[0767] The server uses natural language processing (NLP) techniques to analyze data, understand context and meaning, and extract important decisions and thought processes. This analysis also incorporates data from the emotion engine.

[0768] Step 9:

[0769] The server generates a personality model and thought patterns of the user based on the extracted information. This takes into account the user's emotional state. For example, it identifies situations in which the user is likely to feel stressed and situations in which they are able to make calm judgments.

[0770] Step 10:

[0771] The avatar is designed based on personality models and thought patterns generated by the server. This design also incorporates data from the emotion engine.

[0772] Step 11:

[0773] The device asks the avatar simulation questions to test its functionality. For example, it might ask, "Please explain your new market strategy."

[0774] Step 12:

[0775] The device checks the avatar's response and verifies whether that response meets the user's expectations. The server adjusts the avatar's behavior as needed.

[0776] Step 13:

[0777] The emotion engine monitors the user's emotional state based on the avatar's responses and adjusts the avatar's movements and response tone accordingly. For example, it can be configured to have the avatar respond in a relaxed tone.

[0778] Step 14:

[0779] The device allows the user to finalize the avatar and collects any necessary changes or additional information. This ensures that the avatar behaves as the user expects.

[0780] Step 15:

[0781] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar could give a presentation at a meeting on their behalf.

[0782] Step 16:

[0783] The server monitors the avatar's movements and responses in real time and collects execution logs.

[0784] Step 17:

[0785] The server analyzes the collected logs to evaluate the avatar's performance. This evaluation also includes user emotion data provided by the emotion engine.

[0786] Step 18:

[0787] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance. For example, it might make adjustments to maintain a more relaxed tone in the next presentation.

[0788] Step 19:

[0789] The server periodically collects new information from users and updates avatar data to the latest version. This ensures that avatars always reflect the most up-to-date knowledge and experience.

[0790] Through the steps described above, the system of the present invention reproduces the user's knowledge and experience as an avatar, and by combining it with an emotion engine, it becomes possible to have natural conversations and task responses that take into account the user's emotional state.

[0791] (Example 2)

[0792] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0793] Conventional avatar systems that utilize personal information require not only effective reproduction of the user's knowledge and experience, but also natural dialogue and task execution that takes into account the user's emotional state. However, current technology is insufficient for emotion analysis, making it difficult to adjust responses and actions in response to fluctuations in the user's emotions. Furthermore, performance evaluation of tasks performed by avatars and improvements based on feedback are not effectively carried out, thus failing to improve the quality of the user experience. In addition, cleaning processes and the use of natural language processing technologies are necessary to maintain the quality of collected data.

[0794] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for asking a number of questions to collect an individual's knowledge, thoughts, experiences, and memories; means for analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its movements, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for analyzing emotional data and adjusting the avatar's responses and movements according to the user's emotional state; means for data cleaning to remove noise and misinformation; means for analyzing context and meaning using natural language processing technology and extracting important decisions and thought processes; and means for analyzing the collected emotional data using an emotion engine. This enables natural and effective dialogue and task execution that takes into account not only the user's knowledge and experience but also their emotional state.

[0795] "Personal knowledge, thoughts, experiences, and memories" refers to the specialized knowledge, thought patterns, specific experiences and events, and memories associated with them that an individual possesses.

[0796] "Means of asking questions" refers to devices and programs for presenting questions to users and collecting their answers.

[0797] "Means for analyzing response data in real time" refers to devices and programs that process responses obtained from users immediately and extract and analyze necessary information.

[0798] A "database" refers to a system for organizing and storing collected data.

[0799] "Means for generating personality models and thought patterns" refers to devices and programs that model an individual's personality and thought patterns from collected data.

[0800] An "avatar" refers to a character that acts as a representative of a user in a digital environment.

[0801] "Means for simulating and verifying actions" refers to devices and programs that virtually test the actions of an avatar and evaluate their suitability.

[0802] "Means for performing specific tasks" refers to devices and programs that allow an avatar to perform specific roles or tasks on behalf of the user.

[0803] "Means for collecting execution logs and performing performance evaluation" refers to a device and program that records the avatar's behavior history and evaluates its performance.

[0804] "Means of improving avatar performance based on feedback" refers to devices and programs that improve avatar functionality by incorporating evaluations and opinions from users.

[0805] "Means for analyzing emotional data" refers to devices and programs that recognize a user's emotional state and analyze that data.

[0806] "Data cleaning means" refers to devices and programs that remove noise and misinformation from collected data and improve data quality.

[0807] "Natural language processing technology" refers to the technology used to understand, process, and generate human language using computers.

[0808] "Means for analyzing collected emotional data" refers to devices and programs that process emotional data obtained from users and understand its content.

[0809] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and has it function as a proxy for the user. Furthermore, it integrates an emotion engine that recognizes the user's emotions, enabling more natural dialogue and advanced task handling. This system is implemented through the following basic phases.

[0810] User data collection phase

[0811] The terminal presents the user with an initial question such as, "Please tell me about your work experience." The user enters a response into the terminal, such as, "I have 5 years of experience as a marketing manager." The server analyzes this response data in real time, extracting important keywords and phrases. This data is stored in a structured database. The server then generates additional questions and sends them to the terminal. An emotion engine also operates, analyzing the user's emotional state. For example, it analyzes the user's tone of voice and facial expressions while they are answering to determine whether they are relaxed or nervous.

[0812] Data Analysis and Processing Phase

[0813] The server cleans the collected data, removing noise and misinformation. Using natural language processing (NLP) techniques, it analyzes the context and meaning of the user's writing to extract important decisions and thought processes. This analysis also incorporates data from the emotion engine, taking the user's emotional state into account. Based on the extracted information, the server generates a personality model and thought patterns of the user.

[0814] Avatar generation phase

[0815] The avatar is designed based on personality models and thought patterns generated by the server. Data provided by the emotion engine is also reflected. The terminal verifies the avatar's behavior. For example, it performs a simulation asking, "Please explain the new market strategy," and verifies whether the avatar responds appropriately. During this process, the emotion engine monitors the user's emotional state based on the avatar's responses and makes necessary adjustments.

[0816] Avatar Usage Phase

[0817] When a user is away on a business trip or busy, their device can have an avatar perform specific tasks. For example, the avatar can give a presentation at a meeting. A server monitors the avatar's actions and responses in real time and collects execution logs. The server analyzes the collected logs and evaluates the avatar's performance. This evaluation also includes user emotion data provided by an emotion engine. For example, it checks whether the avatar was nervous during the presentation. The user provides feedback through their device, and the server uses that feedback to improve the avatar's performance.

[0818] Utilizing the Emotion Engine

[0819] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar will respond in a calmer tone. As a concrete example, when a company leader is on a long business trip, the avatar attends internal meetings on their behalf and makes decisions based on the leader's thought process and judgment criteria. In this scenario, the emotion engine detects any tension or stress the leader might be experiencing, and the avatar responds accordingly with a calmer tone.

[0820] Specific example

[0821] The following scenario is an example of a prompt message.

[0822] "This scenario illustrates a situation where a company leader is on a long-term business trip, but their avatar attends an internal meeting and makes decisions based on the leader's thinking and decision-making criteria. During the meeting, an emotion engine analyzes the leader's emotional data to detect tension and stress. Based on this, the avatar provides calm responses, ensuring the meeting runs smoothly."

[0823] Hardware and software to be used

[0824] Hardware: Devices (PCs, smartphones, tablets, etc.), servers

[0825] Software: Natural Language Processing (NLP) engine, emotion engine, database system

[0826] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0827] Step 1:

[0828] Presentation of initial questions

[0829] The device displays an initial question such as, "Please tell us about your work experience." The user enters their answer using the keyboard or touch input. Specifically, a text box appears on the device screen, and the user enters, "I have 5 years of experience as a marketing manager." This input is saved on the device as text data.

[0830] Step 2:

[0831] Collection of user responses

[0832] The device collects user responses and sends them to the server. For example, it might transfer the user's response data, "I have 5 years of experience as a marketing manager," to the server. This transmitted data becomes input data for the server's analysis system.

[0833] Step 3:

[0834] Analysis of response data and keyword extraction

[0835] The server analyzes user response data in real time and extracts important keywords and phrases. Specifically, it uses natural language processing (NLP) algorithms to extract keywords such as "5 years" and "marketing manager." The input data is the user's response text, and the extracted keywords become the output data. This extracted data is structured and stored in a database.

[0836] Step 4:

[0837] Generating and submitting additional questions

[0838] The server generates additional questions based on the extracted keywords and sends them to the terminal. For example, an additional question such as "Please tell us about any specific projects or successful campaigns you have worked on recently" might be generated. The input data is the extracted keywords, and the generated additional questions become the output data.

[0839] Step 5:

[0840] Emotional analysis

[0841] The emotion engine analyzes the user's emotional state. Specifically, it analyzes voice tone and facial expression data to determine whether the user is relaxed or tense. The input data consists of recordings of the user's voice and facial expressions, and the output data is the emotional state (e.g., relaxed, tense).

[0842] Step 6:

[0843] Data Cleaning

[0844] The server cleans the collected data, removing noise and misinformation. For example, it filters out grammatical errors and irrelevant information from the input data. The input data is the user's text responses, and the cleaned text data becomes the output data.

[0845] Step 7:

[0846] Contextual analysis using natural language processing

[0847] The server uses an NLP engine to analyze the context and meaning of the user's text. Specifically, it analyzes text data to extract contextual information such as "the user has had successful marketing experiences." The input data is cleaned text data, and the contextual information becomes the output data.

[0848] Step 8:

[0849] Extraction of important decisions and thought processes

[0850] The server extracts important user decisions and thought processes from the analysis results. For example, it identifies situations where the user feels stressed or situations where they make calm judgments. The input data is the analyzed contextual information, and the characteristics of the thought process become the output data.

[0851] Step 9:

[0852] Generating personality models and thought patterns

[0853] The server generates personality models and thought patterns based on extracted thought processes. For example, it might create a model such as "Users tend to make calm judgments." The input data is the characteristics of the thought process, and the generated personality model becomes the output data.

[0854] Step 10:

[0855] Avatar design

[0856] The server designs the avatar based on the generated personality model and thought patterns. It also incorporates data from the emotion engine. The input data consists of the personality model and emotion data, while the avatar specifications become the output data.

[0857] Step 11:

[0858] Avatar operation check

[0859] The device performs a simulation, such as asking the avatar to "explain the new market strategy," to verify its operation. For example, the device asks a question, and the system checks whether the avatar answers appropriately. The input data is the question used for the simulation, and the response results become the output data.

[0860] Step 12:

[0861] Executing a specific task

[0862] The device instructs an avatar to perform specific tasks, such as giving a presentation. For example, "The avatar explains the marketing strategy for a new product at a meeting." The input data is the task content, and the output data is the result of the execution.

[0863] Step 13:

[0864] Real-time monitoring and log collection

[0865] The server monitors the avatar's actions and responses in real time and collects execution logs. The input data consists of the avatar's actions and responses, while the execution logs are the output data.

[0866] Step 14:

[0867] Performance evaluation and feedback

[0868] The server analyzes collected logs to evaluate avatar performance. It also considers user feedback to improve avatar performance. Input data consists of execution logs and feedback, while the output data represents the improvements made.

[0869] Step 15:

[0870] Adjustment based on emotional data

[0871] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar is set to respond in a calm tone. The input data is emotional data, and the adjusted responses become the output data.

[0872] (Application Example 2)

[0873] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0874] Modern virtual stores require real-time, personalized service to meet user needs for product recommendations and interactive experiences. However, traditional systems struggle to analyze users' emotional states and respond appropriately, sometimes leading to decreased satisfaction. They also struggle to provide personalized product recommendations based on individual user needs. Therefore, there is a need for a system that improves the quality of user interaction and allows users to have an experience in virtual stores that closely resembles that of physical stores.

[0875] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0876] In this invention, the server includes means for asking multiple questions to collect an individual's knowledge, thoughts, experiences, and memories; means for collecting and analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance, and for improving the avatar's performance based on feedback; means for interacting with the user using a smartphone and making product suggestions in a virtual store; and means for analyzing the user's emotional state using an emotion engine and adjusting the avatar's responses. This makes it possible to provide highly personalized product suggestions and services that take into account the user's emotional state.

[0877] "Personal knowledge" refers to the specialized information and experience that individual people possess.

[0878] "Thinking" refers to the process of problem-solving and decision-making that users engage in.

[0879] "Experience" refers to the knowledge and skills that an individual has learned from their past actions and experiences.

[0880] "Memory" refers to an individual's ability to retain past events and information.

[0881] A "question" refers to a question posed to a user in order to obtain information.

[0882] "Response data" refers to the information and responses that users provide in response to questions.

[0883] "Real-time analysis" refers to the process of analyzing data immediately on the spot.

[0884] "Important information" refers to data that is deemed particularly valuable among the analyzed data.

[0885] A "structured database" refers to a database that is organized and stored according to a specific format or set of rules.

[0886] A "personality model" refers to a model that represents the personality traits of individual users.

[0887] "Thinking patterns" refer to the series of actions and decision-making tendencies that users exhibit when solving problems.

[0888] An "avatar" refers to a virtual person or character that acts on behalf of a user.

[0889] "Simulation" refers to the act of virtually recreating a real environment to verify its operation.

[0890] "Performing a specific task" refers to carrying out pre-defined tasks or duties.

[0891] An "execution log" refers to a record of the actions and responses performed by the avatar.

[0892] "Performance evaluation" refers to the process of evaluating the performance of an avatar.

[0893] A "smartphone" refers to a portable, multi-functional information terminal.

[0894] "Dialogue" refers to communication between the user and the system.

[0895] A "virtual store" refers to a virtual sales environment that exists on the internet.

[0896] "Product recommendation" refers to the act of recommending appropriate products to users.

[0897] An "emotion engine" refers to a technology that analyzes a user's emotional state and adjusts its response accordingly.

[0898] "Adjusting your response" means changing your answer to suit the other person's situation and feelings.

[0899] This invention relates to a system that provides personalized product suggestions to users in a virtual store and adjusts responses according to their emotions. Specifically, it provides a system that collects and analyzes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and has the avatar perform specific tasks on behalf of the user. The specific configuration and method for carrying out this invention are described below.

[0900] Hardware and software

[0901] This system consists of a smartphone, a server, a natural language processing engine (NLP), an emotion engine, a database, and a front-end application.

[0902] Hardware: Smartphone (iOS or Android device)

[0903] software:

[0904] Natural Language Processing Engine (NLP): Uses the Google NLP API.

[0905] Emotion engine: Uses Microsoft Azure Emotion API

[0906] Database: Firebase Realtime Database is used.

[0907] Frontend application: Developed using React Native.

[0908] Data collection and analysis

[0909] In the initial stages of the system, the server collects data about the user's knowledge, thoughts, experiences, and memories. This involves a process where the user is asked multiple questions via their smartphone and provides answers. This response data is analyzed in real time by the server, and important information is extracted.

[0910] Data storage and modeling

[0911] The analyzed data is filtered to remove noise and misinformation before being stored in a structured format in the Firebase Realtime Database. Then, natural language processing (NLP) techniques are used to analyze the context and meaning, generating a user personality model and thought patterns.

[0912] Avatar generation and simulation

[0913] The avatar is designed based on personality models and thought patterns generated by the server. The avatar design also incorporates data from an emotion engine to reflect the user's emotional state. Finally, the avatar's movements are simulated and verified.

[0914] Using avatars

[0915] When a user uses a virtual store, an avatar performs specific tasks on their behalf, such as suggesting products. Specifically, it interacts with the user using a smartphone, analyzes the user's emotional state using an emotion engine, and adjusts its response accordingly.

[0916] Feedback and Performance Evaluation

[0917] Avatar execution logs are collected and performance is evaluated. This includes user feedback, and the collected feedback can be used to improve the avatar's performance. This feedback loop allows for continuous improvement of the avatar's behavior and responsiveness.

[0918] Examples of specific cases and prompt statements

[0919] The following interactions are examples of what can be considered.

[0920] Example of a prompt:

[0921] "Please tell me about your work experience."

[0922] User response:

[0923] "I have five years of experience as a marketing manager."

[0924] The data collected in this way is analyzed, and an avatar is generated that interacts with the user in the virtual store based on the user's personality model. This makes it possible to provide highly personalized product suggestions and services that take into account the user's emotional state.

[0925] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0926] Step 1:

[0927] The server initiates a process to collect the user's knowledge, thoughts, experiences, and memories. Specifically, it asks the user multiple questions via their device (smartphone). The user's responses to these questions are collected and analyzed in real time.

[0928] Input: Response data entered by the user on the device.

[0929] Output: Set of collected response data

[0930] Step 2:

[0931] The server analyzes the collected response data in real time and extracts important information. It uses the Google NLP API to analyze the context and meaning of the response data and extract key keywords and phrases.

[0932] Input: Collected response data

[0933] Output: Important keywords and phrases

[0934] Step 3:

[0935] The server stores the extracted key keywords and phrases in a structured database (Firebase Realtime Database). This data will be used for future analysis and avatar generation.

[0936] Input: Important keywords or phrases

[0937] Output: Structured data stored in the database

[0938] Step 4:

[0939] The server generates individual personality models and thought patterns based on stored data. Natural language processing (NLP) techniques are used to model user characteristics and patterns.

[0940] Input: Structured data stored in a database

[0941] Output: Personality model and thought patterns

[0942] Step 5:

[0943] The server designs avatars based on generated personality models and thought patterns. It also integrates data from emotion engines such as the Microsoft Azure Emotion API to consider emotion-based responses.

[0944] Input: Data from personality models, thought patterns, and emotion engines.

[0945] Output: Designed avatar

[0946] Step 6:

[0947] The server simulates and verifies the avatar's movements. This process verifies whether the avatar can respond appropriately and perform specific tasks. Adjustments are made as needed.

[0948] Input: Designed avatar

[0949] Output: Avatars whose operation has been confirmed

[0950] Step 7:

[0951] When a user uses a virtual store, an avatar performs specific tasks on their behalf, such as suggesting products, via their device. The avatar interacts with the user using a smartphone, analyzes the user's emotional state using an emotion engine, and adjusts its responses accordingly.

[0952] Input: User questions, requests, and sentiment data

[0953] Output: Product suggestions and responses

[0954] Step 8:

[0955] The server collects avatar execution logs and performs performance evaluations. Based on the collected data, including user feedback, the avatar's performance is improved.

[0956] Input: Avatar execution log, user feedback

[0957] Output: Evaluation results, improved avatar performance

[0958] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0959] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0960] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0961] [Third Embodiment]

[0962] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0963] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0964] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0965] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0966] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0967] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0968] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0969] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0970] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0971] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0972] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0973] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0974] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. This system is implemented through the following basic phases.

[0975] 1. User data collection phase

[0976] First, the terminal asks the user an initial question. For example, it might ask, "Please tell me about your work experience." The user then enters their answer into the terminal. For example, they might answer, "I have 5 years of experience as a marketing manager." The server then analyzes the user's response data in real time and extracts important keywords and phrases. This analyzed data is immediately stored in a structured database.

[0977] Next, the server generates additional questions based on important information from the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?" The user's additional responses are also saved.

[0978] 2. Data Analysis and Processing Phase

[0979] The collected data is first cleaned by the server to remove noise and misinformation. Then, using natural language processing (NLP) techniques, the context and meaning of the user's writing are analyzed, and important decisions and thought processes are extracted. Based on this information, the server generates a personality model and thought patterns of the user. This allows for a systematic understanding of what kind of decisions the user makes in what situations.

[0980] 3. Avatar Generation Phase

[0981] The server designs an avatar based on a personality model and thought patterns it generates. This designed avatar actively utilizes the user's past data and functions as a representation of the user. Next, the terminal verifies the avatar's operation. For example, it might ask a simulated question such as, "Please explain the new market strategy." The terminal checks whether the avatar's response meets the user's expectations, and the server makes adjustments if necessary.

[0982] 4. Avatar Usage Phase

[0983] When a user is away on a business trip or too busy, the avatar can perform specific tasks on their behalf. For example, the avatar can give a presentation at a meeting as a task set by the user. The device monitors the avatar's actions and responses in real time and sends the execution logs to the server. The server analyzes these logs and evaluates the avatar's performance. Based on this evaluation, the server collects feedback from the user and improves the avatar's performance.

[0984] This allows the user's knowledge and experience to be continuously utilized even when they are absent or after their death. Furthermore, the avatar's performance can be continuously improved based on feedback, always reflecting the latest knowledge and experience.

[0985] As a concrete example, when a company leader is on a long business trip, an avatar can attend internal meetings on their behalf and make decisions based on the leader's thought process and decision-making criteria. This allows the company to maximize the use of the leader's expertise and experience even in their absence.

[0986] As described above, the present invention provides a system that can be widely applied by digitizing an individual's knowledge and experience and reproducing it as an avatar.

[0987] The following describes the processing flow.

[0988] Step 1:

[0989] The device presents the user with an initial question. For example, it might display a question such as, "Please tell us about your work experience."

[0990] Step 2:

[0991] The user enters their response into the device. For example, they might enter, "I have 5 years of experience as a marketing manager."

[0992] Step 3:

[0993] The server analyzes user response data in real time and extracts important keywords and phrases. For example, it might extract "5 years" or "marketing manager."

[0994] Step 4:

[0995] The server generates additional questions based on the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?"

[0996] Step 5:

[0997] The user enters their answer to an additional question on the device. For example, they might answer, "I led the marketing campaign for a new product."

[0998] Step 6:

[0999] The server also analyzes the additional response data, extracts important information, and saves it to the database. It also generates additional questions as needed.

[1000] Step 7:

[1001] The server cleans the collected data, removing noise and misinformation to maintain data quality.

[1002] Step 8:

[1003] The server analyzes the cleaned data using natural language processing (NLP) techniques to understand the context and meaning, and extract important decisions and thought processes.

[1004] Step 9:

[1005] The server generates a user personality model and thought patterns based on the extracted information. This allows for a systematic understanding of how users make decisions in different situations.

[1006] Step 10:

[1007] The avatar is designed based on personality models and thought patterns generated by the server. This process incorporates the user's past data.

[1008] Step 11:

[1009] The device asks the avatar simulation questions to test its functionality. For example, it might ask, "Please explain your new market strategy."

[1010] Step 12:

[1011] The terminal checks the avatar's response and verifies that it is operating according to the specifications. The server makes adjustments as needed.

[1012] Step 13:

[1013] The device performs a final confirmation with the user to verify that there are no problems with the avatar's operation. This ensures that the avatar operates according to the user's expectations.

[1014] Step 14:

[1015] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar could give a presentation at a meeting on their behalf.

[1016] Step 15:

[1017] The server collects avatar execution logs, recording the avatar's actions and responses.

[1018] Step 16:

[1019] The system analyzes the logs collected by the server to evaluate the avatar's performance. This verifies whether the avatar was able to perform the task properly.

[1020] Step 17:

[1021] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance.

[1022] Step 18:

[1023] The server periodically collects new information from users and updates avatar data to the latest version. This ensures that avatars always reflect the most up-to-date knowledge and experience.

[1024] (Example 1)

[1025] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1026] Currently, many companies and individuals are finding it difficult to digitize their knowledge and experience and utilize that data effectively. In particular, problems arise when individuals are unavailable or too busy to provide support based on their experience and knowledge, especially when making important decisions or judgments. Furthermore, removing noise and misinformation from collected data, and improving the accuracy of data analysis, remain challenges.

[1027] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1028] In this invention, the server includes means for presenting information for asking multiple questions; means for collecting and analyzing an individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for using the avatar to perform specific tasks on behalf of the individual; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for cleaning the collected data from noise and misinformation; and means for analyzing the context and meaning of the cleaned data using natural language processing technology and extracting important decisions and thought processes. This makes it possible to effectively digitize an individual's knowledge and experience and support important decision-making and judgment through the avatar. Furthermore, it allows the individual's knowledge and experience to continue to be utilized even when they are absent or busy.

[1029] An "information presentation means" is a device that presents questions or instructions to the user and collects answers or information.

[1030] "Response data" refers to the information or content that users provide or input to information presentation tools.

[1031] "Methods for real-time analysis" refers to the process and technology of immediately analyzing the content of response data as soon as it is obtained and extracting the necessary information.

[1032] A "structured database" refers to a database where the stored data is organized in a specific format so that it can be easily searched and analyzed.

[1033] A "personality model" is a data-based representation of an individual's personality and behavioral characteristics.

[1034] A "thinking pattern" is a model that extracts and represents the characteristics of an individual's judgment criteria and decision-making process.

[1035] A "simulation method" is a means of testing in advance how an avatar will actually behave.

[1036] A "proxy means" is a method by which an avatar performs a specific task on behalf of an individual.

[1037] An "execution log" refers to a detailed record of when an avatar performs a task.

[1038] "Performance evaluation" is a process that evaluates the quality of the avatar's movements and responses based on execution logs.

[1039] "Feedback" refers to evaluations and opinions regarding the results of task execution or the performance of the avatar.

[1040] "Cleaning methods" refer to techniques and processes that remove noise and misinformation from collected data, thereby improving data quality.

[1041] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human natural language.

[1042] "Important decisions and thought processes" refer to the key information and processes involved in making a particular idea or judgment.

[1043] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. This system is implemented through the following basic phases.

[1044] 1. User data collection phase

[1045] First, the terminal asks the user an initial question. For example, it might ask, "Please tell me about your work experience." The user then enters their answer into the terminal. For example, they might answer, "I have five years of experience as a marketing manager." The server then analyzes the user's response data in real time and extracts important keywords and phrases. This analyzed data is immediately stored in a structured database. Next, the server generates additional questions based on the important information from the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects did you lead?" The user's additional response data is also stored in the same way.

[1046] 2. Data Analysis and Processing Phase

[1047] The collected data is first cleaned by the server to remove noise and misinformation. Then, using natural language processing (NLP) techniques, the context and meaning of the user's writing are analyzed, and important decisions and thought processes are extracted. Based on this information, the server generates a personality model and thought patterns of the user. This allows for a systematic understanding of what kind of decisions the user makes in what situations.

[1048] 3. Avatar Generation Phase

[1049] The server designs an avatar based on a personality model and thought patterns it generates. This designed avatar actively utilizes the user's past data and functions as a representation of the user. Next, the terminal verifies the avatar's operation. For example, it might ask a simulated question such as, "Please explain the new market strategy." The terminal checks whether the avatar's response meets the user's expectations, and the server makes adjustments if necessary.

[1050] 4. Avatar Usage Phase

[1051] When a user is away on a business trip or busy, the avatar performs specific tasks on their behalf. For example, the avatar can give a presentation at a meeting as a task set by the user. The terminal monitors the avatar's actions and responses in real time and sends the execution log to the server. The server analyzes this log and evaluates the avatar's performance. Based on this evaluation, feedback from the user is collected, and the server improves the avatar's performance. This makes it possible to continuously utilize the user's knowledge and experience even when the user is absent or after their death. Furthermore, the avatar's performance continuously improves based on feedback, always reflecting the latest knowledge and experience.

[1052] Hardware and software to be used

[1053] Devices: PC, smartphone, tablet

[1054] Servers: High-performance server clusters (e.g., AWS EC2, Google Cloud Platform)

[1055] Software technologies: Natural language processing (e.g., Spacy, BERT), database management systems (e.g., MySQL, PostgreSQL)

[1056] Examples of specific cases and prompt statements

[1057] As a concrete example, consider a scenario where a company leader is on a long business trip, and an avatar attends internal meetings on their behalf, making decisions based on the leader's thought process and decision-making criteria. In this case, the leader can assign the following tasks to the avatar from their business trip location.

[1058] Examples of prompts to input into a generative AI model

[1059] Now, please give a presentation on our new market strategy at today's meeting. Please base your presentation on the following points:

[1060] 1. The importance and potential benefits of new markets

[1061] 2. Our strengths based on competitive analysis

[1062] 3. Specific action plan and timeline

[1063] In response to this prompt, the avatar can deliver a natural presentation based on the leader's past data and thought patterns generated by the system. The terminal monitors the avatar's actions in real time and sends feedback to the server to improve the avatar's performance.

[1064] With the above configuration, the present invention provides a system that can be widely applied by digitizing an individual's knowledge and experience and reproducing it as an avatar.

[1065] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1066] Step 1: User Data Collection Phase

[1067] Specific actions:

[1068] The device asks the user initial questions. For example, it might ask, "Please tell me about your work experience."

[1069] Input: User's response (e.g., "I have 5 years of experience as a marketing manager")

[1070] Data processing: The server analyzes this response in real time and extracts important keywords and phrases.

[1071] Output: Analyzed data (e.g., "5 years", "Marketing Manager")

[1072] Specific actions:

[1073] The server saves the analysis data to the database.

[1074] Input: Analyzed data

[1075] Output: Structured data stored in the database

[1076] Specific actions:

[1077] The server generates additional questions based on the analyzed data and sends them to the terminal.

[1078] Input: Analysis data

[1079] Data processing: Generating additional questions (e.g., "As a marketing manager, what projects did you lead?")

[1080] Output: Additional questions

[1081] Step 2: Data Analysis and Processing Phase

[1082] Specific actions:

[1083] The server cleans the collected data, removing noise and misinformation.

[1084] Input: Collected data

[1085] Data processing: Data cleaning

[1086] Output: Cleaned data

[1087] Specific actions:

[1088] The server uses NLP (Neuro-Linguistic Programming) technology to analyze context and meaning.

[1089] Input: Cleaned data

[1090] Data processing: Contextual analysis through natural language processing (e.g., using the BERT model)

[1091] Output: Information with context and meaning analyzed.

[1092] Specific actions:

[1093] The server generates a user personality model and thought patterns from the analysis results.

[1094] Input: Information analyzed for context and meaning

[1095] Data processing: Generation of personality models and thought patterns

[1096] Output: User personality model and thought patterns

[1097] Step 3: Avatar Generation Phase

[1098] Specific actions:

[1099] The avatar is designed based on personality models and thought patterns generated by the server.

[1100] Input: User's personality model and thought patterns

[1101] Data processing: Avatar design

[1102] Output: Designed avatar

[1103] Specific actions:

[1104] The device will perform a function check on the avatar.

[1105] Input: Designed avatar

[1106] Data calculation: Ask simulation questions and observe the avatar's responses (e.g., "Please explain the new market strategy").

[1107] Output: Avatar response data

[1108] Specific actions:

[1109] The server will adjust the avatar as needed.

[1110] Input: Avatar response data

[1111] Data processing: Confirmation and adjustment of responses

[1112] Output: Adjusted avatar

[1113] Step 4: Avatar Utilization Phase

[1114] Specific actions:

[1115] The avatar performs tasks set by the user.

[1116] Input: User's task settings (e.g., "Presentation at a meeting")

[1117] Data processing: Avatar performs tasks

[1118] Output: Task execution results

[1119] Specific actions:

[1120] The device monitors the avatar's movements and responses in real time and collects logs.

[1121] Input: Avatar movement data

[1122] Data processing: Collection of operation logs

[1123] Output: Collected execution logs

[1124] Specific actions:

[1125] The server analyzes the execution logs and performs a performance evaluation.

[1126] Input: Collected execution logs

[1127] Data processing: Performance evaluation

[1128] Output: Evaluation results and necessary improvements

[1129] Specific actions:

[1130] The server improves avatar performance based on user feedback.

[1131] Input: User feedback and evaluation results

[1132] Data processing: Improving avatar performance based on feedback.

[1133] Output: Improved performance avatar

[1134] (Application Example 1)

[1135] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1136] In recent years, there has been a growing demand for technology that digitizes an individual's knowledge and experience, and uses that data to generate avatars that function as digital representations. However, current technology lacks an effective means of identifying in real time what products and services a user is interested in and recommending appropriate products and services based on those preferences. Furthermore, there is a lack of means to quickly improve the avatar's behavior and responses based on user feedback. Solving these challenges is essential.

[1137] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1138] In this invention, the server includes means for asking multiple questions to collect an individual's knowledge, thoughts, experiences, and memories; means for collecting and analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for collecting the user's real-time movements and gaze via a smart terminal and analyzing the user's preferences and gaze information; and means for generating the user's avatar based on the analyzed information and recommending products and services in real time. This makes it possible to recommend products and services that reflect the user's preferences and gaze in real time.

[1139] "Personal knowledge, thoughts, experiences, and memories" refers to the collection of information, ways of thinking, actual activities and processes performed, and past experiences of a particular individual.

[1140] "Means of asking questions" refers to methods and techniques for presenting a variety of questions in order to obtain information from an individual.

[1141] "Means for collecting and analyzing response data in real time" refers to methods and technologies for quickly collecting individual responses to questions and immediately analyzing that data.

[1142] "Means for extracting important information from analyzed data and storing it in a structured database" refers to methods and techniques for analyzing collected data, identifying key elements, and storing them in a systematic database.

[1143] "Methods for generating personality models and thought patterns" refer to methods and techniques for modeling an individual's personality and thought patterns based on their behavior and responses.

[1144] "Methods for designing avatars" refers to methods and technologies for designing and creating avatars that function as extensions of an individual, based on generated personality models and thought patterns.

[1145] "Means of performing operational checks and making adjustments as necessary" refers to methods and techniques for simulating the movements of a designed avatar, verifying its suitability and functionality, and correcting any problems.

[1146] "Means of performing specific tasks" refers to methods or technologies that enable an avatar to perform specific tasks or activities on behalf of the user.

[1147] "Means for collecting execution logs and performing performance evaluations" refers to methods and technologies for collecting records of tasks performed by avatars and evaluating their results and efficiency.

[1148] "Methods for improving avatar performance based on feedback" refers to methods and technologies for improving the behavior and performance of avatars using feedback from users and systems.

[1149] "Means of collecting users' real-time actions and gaze via smart devices" refers to methods and technologies for instantly collecting users' actions and gaze via smart devices.

[1150] "Means for analyzing user preferences and gaze information" refers to methods and techniques for analyzing collected user preferences and gaze information and extracting useful patterns and trends from that data.

[1151] "Means of recommending products and services" refers to methods and technologies for presenting and recommending the most suitable products and services to users based on analyzed information.

[1152] This invention relates to a system that collects and analyzes the knowledge, thoughts, experiences, and memories of a specific individual and generates an avatar based on that information. This system is implemented using the following hardware and software.

[1153] Hardware and software to be used

[1154] Hardware: Smart glasses, webcam, server

[1155] Software: Python, OpenCV, NLP module, product recommender engine, avatar generation module

[1156] System Description

[1157] 1. User data collection phase

[1158] The server collects real-time user activity and gaze information via smart glasses. Specifically, it captures video data from the smart glasses using OpenCV and performs analysis to identify the user's gaze and interests.

[1159] 2. Data Analysis Phase

[1160] The server analyzes the collected data using NLP modules and other analytical tools. This identifies products that users are interested in and their preferences. It also performs noise reduction and misinformation cleaning to extract important data.

[1161] 3. Avatar Generation Phase

[1162] Based on the analyzed data, a user personality model and thought patterns are generated, and an avatar is created based on these. Using the avatar generation module, the avatar, which functions as a digital representation of the user, is designed in detail. This designed avatar undergoes functional verification through simulated questions based on the user's gaze and preferences.

[1163] 4. Product Recommendation Phase

[1164] The avatar provides real-time product recommendations. As the user moves around the store wearing smart glasses, the avatar uses its recommendation engine to suggest appropriate products based on their gaze. For example, if the user looks towards the electronics section, the avatar will recommend the most suitable electronic device.

[1165] 5. Feedback Phase

[1166] We receive feedback from users about the avatar's movements and responses, and improve the avatar's performance based on that feedback. We analyze user comments and activity logs and make necessary improvements.

[1167] Specific examples of processing steps

[1168] 1. The user puts on smart glasses and enters the store.

[1169] 2. The server obtains real-time video from the smart glasses and analyzes the user's gaze using OpenCV.

[1170] 3. When a user fixates their gaze on a specific product, an NLP module analyzes their preferences and interests based on that information.

[1171] 4. An avatar is generated, and a recommender engine is used to recommend products based on the analysis results.

[1172] 5. Users provide feedback on recommended products, and the server uses that information to improve the avatar's behavior.

[1173] Example of a prompt

[1174] "Please provide the following data to the NLP Processor to analyze user preferences: {user data}"

[1175] "Provide the Recommender engine with a {product list} and recommend the best products for the user."

[1176] In this way, it becomes possible to realize product recommendations that reflect the user's real-time behavior and gaze information, and to provide advanced personalized services using avatars.

[1177] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1178] Step 1:

[1179] The user wears smart glasses, and the server acquires real-time video from the smart glasses. The input is video data from the smart glasses, and the output is video frames that are processed in real time. The server uses this frame data to detect the user's gaze and analyze the gaze information.

[1180] Step 2:

[1181] The server analyzes the acquired video data using OpenCV to identify the user's gaze information. The input is real-time video frames, and the output is the user's gaze coordinate data. Based on this gaze coordinate data, the server detects objects and products that the user may be interested in.

[1182] Step 3:

[1183] The server collects basic information related to the detected products and objects. The input is a list of products detected based on gaze coordinate data, and the output is basic information related to these products (e.g., product name, category, price, etc.). The server retrieves this information and proceeds to the next analysis stage.

[1184] Step 4:

[1185] The server uses an NLP module to analyze user response data and past purchase history to identify user preferences and interests. The input is user response data and purchase history data, and the output is profile data indicating the user's preferences and interests. Based on this profile data, the server narrows down the list of product recommendation candidates.

[1186] Step 5:

[1187] The server uses a recommender engine to suggest appropriate products based on the user's profile data and gaze information. The input is profile data and gaze coordinate data, and the output is a list of recommended products. The server generates the list of recommended products, and the avatar presents it to the user.

[1188] Step 6:

[1189] The avatar presents the user with a list of recommended products and provides detailed explanations and advice. The input is the list of recommended products, and the output is a product recommendation message to the user. The server delivers this message to the user in real time through the avatar.

[1190] Step 7:

[1191] The system provides feedback on recommended products from the user. The input is user feedback data, and the output is feedback evaluation data as an analysis result. The server collects this feedback data and uses it to inform the avatar's next actions.

[1192] Step 8:

[1193] The server uses feedback evaluation data to make improvements to enhance the avatar's performance. The input is feedback evaluation data, and the output is the improved avatar behavior model. The server applies these improvements to the avatar to prepare for the next interaction.

[1194] This will allow users to receive product recommendations in real time and enjoy personalized services through avatars.

[1195] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1196] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, more natural dialogue and advanced task handling become possible. This system is implemented through the following basic phases.

[1197] 1. User data collection phase

[1198] First, the device presents the user with an initial question. For example, it might display a question such as, "Please tell me about your work experience." The user then enters their answer into the device. For example, they might enter an answer such as, "I have 5 years of experience as a marketing manager."

[1199] The server analyzes user response data in real time and extracts important keywords and phrases. For example, information such as "5 years" and "marketing manager" may be extracted. This data is stored in a structured database.

[1200] Next, the server generates additional questions based on the important information and sends them to the terminal. The emotion engine also operates simultaneously, recognizing the user's emotional state. For example, the emotion engine analyzes the user's tone of voice and facial expressions while they are answering to determine whether the user is relaxed or tense.

[1201] 2. Data Analysis and Processing Phase

[1202] The collected data is cleaned by the server to remove noise and misinformation. Next, natural language processing (NLP) techniques are used to analyze the context and meaning of the user's writing and extract important decisions and thought processes. This analysis also incorporates data from an emotion engine, taking the user's emotional state into consideration.

[1203] Based on the extracted information, the server generates a personality model and thought patterns of the user. For example, it identifies situations in which the user is likely to feel stressed and situations in which they can make calm judgments.

[1204] 3. Avatar Generation Phase

[1205] The avatar is designed based on personality models and thought patterns generated by the server. Data provided by the emotion engine is also incorporated into this process. The terminal then verifies the avatar's behavior. For example, as a simulation, it might ask the question, "Please explain the new market strategy," and verify whether the avatar responds appropriately.

[1206] In this process, the emotion engine monitors the user's emotional state based on the avatar's responses and makes necessary adjustments. For example, if the user answers a question in a relaxed tone, the avatar will be adjusted to respond in a similar tone.

[1207] 4. Avatar Usage Phase

[1208] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar can give a presentation at a meeting. The server monitors the avatar's actions and responses in real time and collects execution logs.

[1209] The server analyzes the collected logs to evaluate the avatar's performance. This evaluation also includes user emotion data provided by the emotion engine. For example, it checks whether the avatar was nervous during the presentation.

[1210] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance. For example, it might make adjustments to maintain a more relaxed tone in the next presentation.

[1211] 5. Utilizing the Emotional Engine

[1212] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar is set to respond in a calmer tone. This makes communication with the user more natural and effective.

[1213] As a concrete example, when a company leader is on a long business trip, an avatar can attend internal meetings on their behalf and make decisions based on the leader's thought process and decision-making criteria. In this scenario, an emotional engine detects any tension or stress the leader might feel during the meeting, and the avatar responds calmly to address these issues, ensuring the meeting proceeds smoothly.

[1214] As described above, the present invention provides a system that digitizes an individual's knowledge and experience and combines it with an emotion engine to reproduce it as an avatar. This makes it possible to continuously utilize the user's knowledge and experience even when the user is absent or after death, and the emotion engine makes communication more natural and effective.

[1215] The following describes the processing flow.

[1216] Step 1:

[1217] The device presents the user with an initial question. For example, it might display a question such as, "Please tell us about your work experience."

[1218] Step 2:

[1219] The user enters their answer into the device. For example, they might enter, "I have 5 years of experience as a marketing manager."

[1220] Step 3:

[1221] The server analyzes user response data in real time and extracts important keywords and phrases. For example, it might extract "5 years" or "marketing manager."

[1222] Step 4:

[1223] The device uses an emotion engine to recognize the user's emotions in real time. For example, it analyzes the user's voice tone and facial expressions to determine whether the user is relaxed or stressed.

[1224] Step 5:

[1225] The server generates additional questions based on the analyzed data and the sentiment engine data, and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?"

[1226] Step 6:

[1227] The user enters their answer to an additional question on the device. For example, they might answer, "I led the marketing campaign for a new product."

[1228] Step 7:

[1229] The server also analyzes the additional response data, extracts important information, and saves it to the database. It also performs cleaning processes to remove noise and misinformation.

[1230] Step 8:

[1231] The server uses natural language processing (NLP) techniques to analyze data, understand context and meaning, and extract important decisions and thought processes. This analysis also incorporates data from the emotion engine.

[1232] Step 9:

[1233] The server generates a personality model and thought patterns of the user based on the extracted information. This takes into account the user's emotional state. For example, it identifies situations in which the user is likely to feel stressed and situations in which they are able to make calm judgments.

[1234] Step 10:

[1235] The avatar is designed based on personality models and thought patterns generated by the server. This design also incorporates data from the emotion engine.

[1236] Step 11:

[1237] The device asks the avatar simulation questions to test its functionality. For example, it might ask, "Please explain your new market strategy."

[1238] Step 12:

[1239] The device checks the avatar's response and verifies whether that response meets the user's expectations. The server adjusts the avatar's behavior as needed.

[1240] Step 13:

[1241] The emotion engine monitors the user's emotional state based on the avatar's responses and adjusts the avatar's movements and response tone accordingly. For example, it can be configured to have the avatar respond in a relaxed tone.

[1242] Step 14:

[1243] The device allows the user to finalize the avatar and collects any necessary changes or additional information. This ensures that the avatar behaves as the user expects.

[1244] Step 15:

[1245] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar could give a presentation at a meeting on their behalf.

[1246] Step 16:

[1247] The server monitors the avatar's movements and responses in real time and collects execution logs.

[1248] Step 17:

[1249] The server analyzes the collected logs to evaluate the avatar's performance. This evaluation also includes user emotion data provided by the emotion engine.

[1250] Step 18:

[1251] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance. For example, it might make adjustments to maintain a more relaxed tone in the next presentation.

[1252] Step 19:

[1253] The server periodically collects new information from users and updates avatar data to the latest version. This ensures that avatars always reflect the most up-to-date knowledge and experience.

[1254] Through the steps described above, the system of the present invention reproduces the user's knowledge and experience as an avatar, and by combining it with an emotion engine, it becomes possible to have natural conversations and task responses that take into account the user's emotional state.

[1255] (Example 2)

[1256] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1257] Conventional avatar systems that utilize personal information require not only effective reproduction of the user's knowledge and experience, but also natural dialogue and task execution that takes into account the user's emotional state. However, current technology is insufficient for emotion analysis, making it difficult to adjust responses and actions in response to fluctuations in the user's emotions. Furthermore, performance evaluation of tasks performed by avatars and improvements based on feedback are not effectively carried out, thus failing to improve the quality of the user experience. In addition, cleaning processes and the use of natural language processing technologies are necessary to maintain the quality of collected data.

[1258] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for asking a number of questions to collect an individual's knowledge, thoughts, experiences, and memories; means for analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its movements, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for analyzing emotional data and adjusting the avatar's responses and movements according to the user's emotional state; means for data cleaning to remove noise and misinformation; means for analyzing context and meaning using natural language processing technology and extracting important decisions and thought processes; and means for analyzing the collected emotional data using an emotion engine. This enables natural and effective dialogue and task execution that takes into account not only the user's knowledge and experience but also their emotional state.

[1259] "Personal knowledge, thoughts, experiences, and memories" refers to the specialized knowledge, thought patterns, specific experiences and events, and memories associated with them that an individual possesses.

[1260] "Means of asking questions" refers to devices and programs for presenting questions to users and collecting their answers.

[1261] "Means for analyzing response data in real time" refers to devices and programs that process responses obtained from users immediately and extract and analyze necessary information.

[1262] A "database" refers to a system for organizing and storing collected data.

[1263] "Means for generating personality models and thought patterns" refers to devices and programs that model an individual's personality and thought patterns from collected data.

[1264] An "avatar" refers to a character that acts as a representative of a user in a digital environment.

[1265] "Means for simulating and verifying actions" refers to devices and programs that virtually test the actions of an avatar and evaluate their suitability.

[1266] "Means for performing specific tasks" refers to devices and programs that allow an avatar to perform specific roles or tasks on behalf of the user.

[1267] "Means for collecting execution logs and performing performance evaluation" refers to a device and program that records the avatar's behavior history and evaluates its performance.

[1268] "Means of improving avatar performance based on feedback" refers to devices and programs that improve avatar functionality by incorporating evaluations and opinions from users.

[1269] "Means for analyzing emotional data" refers to devices and programs that recognize a user's emotional state and analyze that data.

[1270] "Data cleaning means" refers to devices and programs that remove noise and misinformation from collected data and improve data quality.

[1271] "Natural language processing technology" refers to the technology used to understand, process, and generate human language using computers.

[1272] "Means for analyzing collected emotional data" refers to devices and programs that process emotional data obtained from users and understand its content.

[1273] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and has it function as a proxy for the user. Furthermore, it integrates an emotion engine that recognizes the user's emotions, enabling more natural dialogue and advanced task handling. This system is implemented through the following basic phases.

[1274] User data collection phase

[1275] The terminal presents the user with an initial question such as, "Please tell me about your work experience." The user enters a response into the terminal, such as, "I have 5 years of experience as a marketing manager." The server analyzes this response data in real time, extracting important keywords and phrases. This data is stored in a structured database. The server then generates additional questions and sends them to the terminal. An emotion engine also operates, analyzing the user's emotional state. For example, it analyzes the user's tone of voice and facial expressions while they are answering to determine whether they are relaxed or nervous.

[1276] Data Analysis and Processing Phase

[1277] The server cleans the collected data, removing noise and misinformation. Using natural language processing (NLP) techniques, it analyzes the context and meaning of the user's writing to extract important decisions and thought processes. This analysis also incorporates data from the emotion engine, taking the user's emotional state into account. Based on the extracted information, the server generates a personality model and thought patterns of the user.

[1278] Avatar generation phase

[1279] The avatar is designed based on personality models and thought patterns generated by the server. Data provided by the emotion engine is also reflected. The terminal verifies the avatar's behavior. For example, it performs a simulation asking, "Please explain the new market strategy," and verifies whether the avatar responds appropriately. During this process, the emotion engine monitors the user's emotional state based on the avatar's responses and makes necessary adjustments.

[1280] Avatar Usage Phase

[1281] When a user is away on a business trip or busy, their device can have an avatar perform specific tasks. For example, the avatar can give a presentation at a meeting. A server monitors the avatar's actions and responses in real time and collects execution logs. The server analyzes the collected logs and evaluates the avatar's performance. This evaluation also includes user emotion data provided by an emotion engine. For example, it checks whether the avatar was nervous during the presentation. The user provides feedback through their device, and the server uses that feedback to improve the avatar's performance.

[1282] Utilizing the Emotion Engine

[1283] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar will respond in a calmer tone. As a concrete example, when a company leader is on a long business trip, the avatar attends internal meetings on their behalf and makes decisions based on the leader's thought process and judgment criteria. In this scenario, the emotion engine detects any tension or stress the leader might be experiencing, and the avatar responds accordingly with a calmer tone.

[1284] Specific example

[1285] The following scenario is an example of a prompt message.

[1286] "This scenario illustrates a situation where a company leader is on a long-term business trip, but their avatar attends an internal meeting and makes decisions based on the leader's thinking and decision-making criteria. During the meeting, an emotion engine analyzes the leader's emotional data to detect tension and stress. Based on this, the avatar provides calm responses, ensuring the meeting runs smoothly."

[1287] Hardware and software to be used

[1288] Hardware: Devices (PCs, smartphones, tablets, etc.), servers

[1289] Software: Natural Language Processing (NLP) engine, emotion engine, database system

[1290] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1291] Step 1:

[1292] Presentation of initial questions

[1293] The device displays an initial question such as, "Please tell us about your work experience." The user enters their answer using the keyboard or touch input. Specifically, a text box appears on the device screen, and the user enters, "I have 5 years of experience as a marketing manager." This input is saved on the device as text data.

[1294] Step 2:

[1295] Collection of user responses

[1296] The device collects user responses and sends them to the server. For example, it might transfer the user's response data, "I have 5 years of experience as a marketing manager," to the server. This transmitted data becomes input data for the server's analysis system.

[1297] Step 3:

[1298] Analysis of response data and keyword extraction

[1299] The server analyzes user response data in real time and extracts important keywords and phrases. Specifically, it uses natural language processing (NLP) algorithms to extract keywords such as "5 years" and "marketing manager." The input data is the user's response text, and the extracted keywords become the output data. This extracted data is structured and stored in a database.

[1300] Step 4:

[1301] Generating and submitting additional questions

[1302] The server generates additional questions based on the extracted keywords and sends them to the terminal. For example, an additional question such as "Please tell us about any specific projects or successful campaigns you have worked on recently" might be generated. The input data is the extracted keywords, and the generated additional questions become the output data.

[1303] Step 5:

[1304] Emotional analysis

[1305] The emotion engine analyzes the user's emotional state. Specifically, it analyzes voice tone and facial expression data to determine whether the user is relaxed or tense. The input data consists of recordings of the user's voice and facial expressions, and the output data is the emotional state (e.g., relaxed, tense).

[1306] Step 6:

[1307] Data Cleaning

[1308] The server cleans the collected data, removing noise and misinformation. For example, it filters out grammatical errors and irrelevant information from the input data. The input data is the user's text responses, and the cleaned text data becomes the output data.

[1309] Step 7:

[1310] Contextual analysis using natural language processing

[1311] The server uses an NLP engine to analyze the context and meaning of the user's text. Specifically, it analyzes text data to extract contextual information such as "the user has had successful marketing experiences." The input data is cleaned text data, and the contextual information becomes the output data.

[1312] Step 8:

[1313] Extraction of important decisions and thought processes

[1314] The server extracts important user decisions and thought processes from the analysis results. For example, it identifies situations where the user feels stressed or situations where they make calm judgments. The input data is the analyzed contextual information, and the characteristics of the thought process become the output data.

[1315] Step 9:

[1316] Generating personality models and thought patterns

[1317] The server generates personality models and thought patterns based on extracted thought processes. For example, it might create a model such as "Users tend to make calm judgments." The input data is the characteristics of the thought process, and the generated personality model becomes the output data.

[1318] Step 10:

[1319] Avatar design

[1320] The server designs the avatar based on the generated personality model and thought patterns. It also incorporates data from the emotion engine. The input data consists of the personality model and emotion data, while the avatar specifications become the output data.

[1321] Step 11:

[1322] Avatar operation check

[1323] The device performs a simulation, such as asking the avatar to "explain the new market strategy," to verify its operation. For example, the device asks a question, and the system checks whether the avatar answers appropriately. The input data is the question used for the simulation, and the response results become the output data.

[1324] Step 12:

[1325] Executing a specific task

[1326] The device instructs an avatar to perform specific tasks, such as giving a presentation. For example, "The avatar explains the marketing strategy for a new product at a meeting." The input data is the task content, and the output data is the result of the execution.

[1327] Step 13:

[1328] Real-time monitoring and log collection

[1329] The server monitors the avatar's actions and responses in real time and collects execution logs. The input data consists of the avatar's actions and responses, while the execution logs are the output data.

[1330] Step 14:

[1331] Performance evaluation and feedback

[1332] The server analyzes collected logs to evaluate avatar performance. It also considers user feedback to improve avatar performance. Input data consists of execution logs and feedback, while the output data represents the improvements made.

[1333] Step 15:

[1334] Adjustment based on emotional data

[1335] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar is set to respond in a calm tone. The input data is emotional data, and the adjusted responses become the output data.

[1336] (Application Example 2)

[1337] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1338] Modern virtual stores require real-time, personalized service to meet user needs for product recommendations and interactive experiences. However, traditional systems struggle to analyze users' emotional states and respond appropriately, sometimes leading to decreased satisfaction. They also struggle to provide personalized product recommendations based on individual user needs. Therefore, there is a need for a system that improves the quality of user interaction and allows users to have an experience in virtual stores that closely resembles that of physical stores.

[1339] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1340] In this invention, the server includes means for asking multiple questions to collect an individual's knowledge, thoughts, experiences, and memories; means for collecting and analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance, and for improving the avatar's performance based on feedback; means for interacting with the user using a smartphone and making product suggestions in a virtual store; and means for analyzing the user's emotional state using an emotion engine and adjusting the avatar's responses. This makes it possible to provide highly personalized product suggestions and services that take into account the user's emotional state.

[1341] "Personal knowledge" refers to the specialized information and experience that individual people possess.

[1342] "Thinking" refers to the process of problem-solving and decision-making that users engage in.

[1343] "Experience" refers to the knowledge and skills that an individual has learned from their past actions and experiences.

[1344] "Memory" refers to an individual's ability to retain past events and information.

[1345] A "question" refers to a question posed to a user in order to obtain information.

[1346] "Response data" refers to the information and responses that users provide in response to questions.

[1347] "Real-time analysis" refers to the process of analyzing data immediately on the spot.

[1348] "Important information" refers to data that is deemed particularly valuable among the analyzed data.

[1349] A "structured database" refers to a database that is organized and stored according to a specific format or set of rules.

[1350] A "personality model" refers to a model that represents the personality traits of individual users.

[1351] "Thinking patterns" refer to the series of actions and decision-making tendencies that users exhibit when solving problems.

[1352] An "avatar" refers to a virtual person or character that acts on behalf of a user.

[1353] "Simulation" refers to the act of virtually recreating a real environment to verify its operation.

[1354] "Performing a specific task" refers to carrying out pre-defined tasks or duties.

[1355] An "execution log" refers to a record of the actions and responses performed by the avatar.

[1356] "Performance evaluation" refers to the process of evaluating the performance of an avatar.

[1357] A "smartphone" refers to a portable, multi-functional information terminal.

[1358] "Dialogue" refers to communication between the user and the system.

[1359] A "virtual store" refers to a virtual sales environment that exists on the internet.

[1360] "Product recommendation" refers to the act of recommending appropriate products to users.

[1361] An "emotion engine" refers to a technology that analyzes a user's emotional state and adjusts its response accordingly.

[1362] "Adjusting your response" means changing your answer to suit the other person's situation and feelings.

[1363] This invention relates to a system that provides personalized product suggestions to users in a virtual store and adjusts responses according to their emotions. Specifically, it provides a system that collects and analyzes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and has the avatar perform specific tasks on behalf of the user. The specific configuration and method for carrying out this invention are described below.

[1364] Hardware and software

[1365] This system consists of a smartphone, a server, a natural language processing engine (NLP), an emotion engine, a database, and a front-end application.

[1366] Hardware: Smartphone (iOS or Android device)

[1367] software:

[1368] Natural Language Processing Engine (NLP): Uses the Google NLP API.

[1369] Emotion engine: Uses Microsoft Azure Emotion API

[1370] Database: Firebase Realtime Database is used.

[1371] Frontend application: Developed using React Native.

[1372] Data collection and analysis

[1373] In the initial stages of the system, the server collects data about the user's knowledge, thoughts, experiences, and memories. This involves a process where the user is asked multiple questions via their smartphone and provides answers. This response data is analyzed in real time by the server, and important information is extracted.

[1374] Data storage and modeling

[1375] The analyzed data is filtered to remove noise and misinformation before being stored in a structured format in the Firebase Realtime Database. Then, natural language processing (NLP) techniques are used to analyze the context and meaning, generating a user personality model and thought patterns.

[1376] Avatar generation and simulation

[1377] The avatar is designed based on personality models and thought patterns generated by the server. The avatar design also incorporates data from an emotion engine to reflect the user's emotional state. Finally, the avatar's movements are simulated and verified.

[1378] Using avatars

[1379] When a user uses a virtual store, an avatar performs specific tasks on their behalf, such as suggesting products. Specifically, it interacts with the user using a smartphone, analyzes the user's emotional state using an emotion engine, and adjusts its response accordingly.

[1380] Feedback and Performance Evaluation

[1381] Avatar execution logs are collected and performance is evaluated. This includes user feedback, and the collected feedback can be used to improve the avatar's performance. This feedback loop allows for continuous improvement of the avatar's behavior and responsiveness.

[1382] Examples of specific cases and prompt statements

[1383] The following interactions are examples of what can be considered.

[1384] Example of a prompt:

[1385] "Please tell me about your work experience."

[1386] User response:

[1387] "I have five years of experience as a marketing manager."

[1388] The data collected in this way is analyzed, and an avatar is generated that interacts with the user in the virtual store based on the user's personality model. This makes it possible to provide highly personalized product suggestions and services that take into account the user's emotional state.

[1389] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1390] Step 1:

[1391] The server initiates a process to collect the user's knowledge, thoughts, experiences, and memories. Specifically, it asks the user multiple questions via their device (smartphone). The user's responses to these questions are collected and analyzed in real time.

[1392] Input: Response data entered by the user on the device.

[1393] Output: Set of collected response data

[1394] Step 2:

[1395] The server analyzes the collected response data in real time and extracts important information. It uses the Google NLP API to analyze the context and meaning of the response data and extract key keywords and phrases.

[1396] Input: Collected response data

[1397] Output: Important keywords and phrases

[1398] Step 3:

[1399] The server stores the extracted key keywords and phrases in a structured database (Firebase Realtime Database). This data will be used for future analysis and avatar generation.

[1400] Input: Important keywords or phrases

[1401] Output: Structured data stored in the database

[1402] Step 4:

[1403] The server generates individual personality models and thought patterns based on stored data. Natural language processing (NLP) techniques are used to model user characteristics and patterns.

[1404] Input: Structured data stored in a database

[1405] Output: Personality model and thought patterns

[1406] Step 5:

[1407] The server designs avatars based on generated personality models and thought patterns. It also integrates data from emotion engines such as the Microsoft Azure Emotion API to consider emotion-based responses.

[1408] Input: Data from personality models, thought patterns, and emotion engines.

[1409] Output: Designed avatar

[1410] Step 6:

[1411] The server simulates and verifies the avatar's movements. This process verifies whether the avatar can respond appropriately and perform specific tasks. Adjustments are made as needed.

[1412] Input: Designed avatar

[1413] Output: Avatars whose operation has been confirmed

[1414] Step 7:

[1415] When a user uses a virtual store, an avatar performs specific tasks on their behalf, such as suggesting products, via their device. The avatar interacts with the user using a smartphone, analyzes the user's emotional state using an emotion engine, and adjusts its responses accordingly.

[1416] Input: User questions, requests, and sentiment data

[1417] Output: Product suggestions and responses

[1418] Step 8:

[1419] The server collects avatar execution logs and performs performance evaluations. Based on the collected data, including user feedback, the avatar's performance is improved.

[1420] Input: Avatar execution log, user feedback

[1421] Output: Evaluation results, improved avatar performance

[1422] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1423] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1424] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1425] [Fourth Embodiment]

[1426] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1427] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1428] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1429] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1430] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1431] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1432] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1433] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1434] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1435] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1436] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1437] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1438] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1439] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. This system is implemented through the following basic phases.

[1440] 1. User data collection phase

[1441] First, the terminal asks the user an initial question. For example, it might ask, "Please tell me about your work experience." The user then enters their answer into the terminal. For example, they might answer, "I have 5 years of experience as a marketing manager." The server then analyzes the user's response data in real time and extracts important keywords and phrases. This analyzed data is immediately stored in a structured database.

[1442] Next, the server generates additional questions based on important information from the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?" The user's additional responses are also saved.

[1443] 2. Data Analysis and Processing Phase

[1444] The collected data is first cleaned by the server to remove noise and misinformation. Then, using natural language processing (NLP) techniques, the context and meaning of the user's writing are analyzed, and important decisions and thought processes are extracted. Based on this information, the server generates a personality model and thought patterns of the user. This allows for a systematic understanding of what kind of decisions the user makes in what situations.

[1445] 3. Avatar Generation Phase

[1446] The server designs an avatar based on a personality model and thought patterns it generates. This designed avatar actively utilizes the user's past data and functions as a representation of the user. Next, the terminal verifies the avatar's operation. For example, it might ask a simulated question such as, "Please explain the new market strategy." The terminal checks whether the avatar's response meets the user's expectations, and the server makes adjustments if necessary.

[1447] 4. Avatar Usage Phase

[1448] When a user is away on a business trip or too busy, the avatar can perform specific tasks on their behalf. For example, the avatar can give a presentation at a meeting as a task set by the user. The device monitors the avatar's actions and responses in real time and sends the execution logs to the server. The server analyzes these logs and evaluates the avatar's performance. Based on this evaluation, the server collects feedback from the user and improves the avatar's performance.

[1449] This allows the user's knowledge and experience to be continuously utilized even when they are absent or after their death. Furthermore, the avatar's performance can be continuously improved based on feedback, always reflecting the latest knowledge and experience.

[1450] As a concrete example, when a company leader is on a long business trip, an avatar can attend internal meetings on their behalf and make decisions based on the leader's thought process and decision-making criteria. This allows the company to maximize the use of the leader's expertise and experience even in their absence.

[1451] As described above, the present invention provides a system that can be widely applied by digitizing an individual's knowledge and experience and reproducing it as an avatar.

[1452] The following describes the processing flow.

[1453] Step 1:

[1454] The device presents the user with an initial question. For example, it might display a question such as, "Please tell us about your work experience."

[1455] Step 2:

[1456] The user enters their response into the device. For example, they might enter, "I have 5 years of experience as a marketing manager."

[1457] Step 3:

[1458] The server analyzes user response data in real time and extracts important keywords and phrases. For example, it might extract "5 years" or "marketing manager."

[1459] Step 4:

[1460] The server generates additional questions based on the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?"

[1461] Step 5:

[1462] The user enters their answer to an additional question on the device. For example, they might answer, "I led the marketing campaign for a new product."

[1463] Step 6:

[1464] The server also analyzes the additional response data, extracts important information, and saves it to the database. It also generates additional questions as needed.

[1465] Step 7:

[1466] The server cleans the collected data, removing noise and misinformation to maintain data quality.

[1467] Step 8:

[1468] The server analyzes the cleaned data using natural language processing (NLP) techniques to understand the context and meaning, and extract important decisions and thought processes.

[1469] Step 9:

[1470] The server generates a user personality model and thought patterns based on the extracted information. This allows for a systematic understanding of how users make decisions in different situations.

[1471] Step 10:

[1472] The avatar is designed based on personality models and thought patterns generated by the server. This process incorporates the user's past data.

[1473] Step 11:

[1474] The device asks the avatar simulation questions to test its functionality. For example, it might ask, "Please explain your new market strategy."

[1475] Step 12:

[1476] The terminal checks the avatar's response and verifies that it is operating according to the specifications. The server makes adjustments as needed.

[1477] Step 13:

[1478] The device performs a final confirmation with the user to verify that there are no problems with the avatar's operation. This ensures that the avatar operates according to the user's expectations.

[1479] Step 14:

[1480] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar could give a presentation at a meeting on their behalf.

[1481] Step 15:

[1482] The server collects avatar execution logs, recording the avatar's actions and responses.

[1483] Step 16:

[1484] The system analyzes the logs collected by the server to evaluate the avatar's performance. This verifies whether the avatar was able to perform the task properly.

[1485] Step 17:

[1486] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance.

[1487] Step 18:

[1488] The server periodically collects new information from users and updates avatar data to the latest version. This ensures that avatars always reflect the most up-to-date knowledge and experience.

[1489] (Example 1)

[1490] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1491] Currently, many companies and individuals are finding it difficult to digitize their knowledge and experience and utilize that data effectively. In particular, problems arise when individuals are unavailable or too busy to provide support based on their experience and knowledge, especially when making important decisions or judgments. Furthermore, removing noise and misinformation from collected data, and improving the accuracy of data analysis, remain challenges.

[1492] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1493] In this invention, the server includes means for presenting information for asking multiple questions; means for collecting and analyzing an individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for using the avatar to perform specific tasks on behalf of the individual; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for cleaning the collected data from noise and misinformation; and means for analyzing the context and meaning of the cleaned data using natural language processing technology and extracting important decisions and thought processes. This makes it possible to effectively digitize an individual's knowledge and experience and support important decision-making and judgment through the avatar. Furthermore, it allows the individual's knowledge and experience to continue to be utilized even when they are absent or busy.

[1494] An "information presentation means" is a device that presents questions or instructions to the user and collects answers or information.

[1495] "Response data" refers to the information or content that users provide or input to information presentation tools.

[1496] "Methods for real-time analysis" refers to the process and technology of immediately analyzing the content of response data as soon as it is obtained and extracting the necessary information.

[1497] A "structured database" refers to a database where the stored data is organized in a specific format so that it can be easily searched and analyzed.

[1498] A "personality model" is a data-based representation of an individual's personality and behavioral characteristics.

[1499] A "thinking pattern" is a model that extracts and represents the characteristics of an individual's judgment criteria and decision-making process.

[1500] A "simulation method" is a means of testing in advance how an avatar will actually behave.

[1501] A "proxy means" is a method by which an avatar performs a specific task on behalf of an individual.

[1502] An "execution log" refers to a detailed record of when an avatar performs a task.

[1503] "Performance evaluation" is a process that evaluates the quality of the avatar's movements and responses based on execution logs.

[1504] "Feedback" refers to evaluations and opinions regarding the results of task execution or the performance of the avatar.

[1505] "Cleaning methods" refer to techniques and processes that remove noise and misinformation from collected data, thereby improving data quality.

[1506] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human natural language.

[1507] "Important decisions and thought processes" refer to the key information and processes involved in making a particular idea or judgment.

[1508] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. This system is implemented through the following basic phases.

[1509] 1. User data collection phase

[1510] First, the terminal asks the user an initial question. For example, it might ask, "Please tell me about your work experience." The user then enters their answer into the terminal. For example, they might answer, "I have five years of experience as a marketing manager." The server then analyzes the user's response data in real time and extracts important keywords and phrases. This analyzed data is immediately stored in a structured database. Next, the server generates additional questions based on the important information from the analyzed data and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects did you lead?" The user's additional response data is also stored in the same way.

[1511] 2. Data Analysis and Processing Phase

[1512] The collected data is first cleaned by the server to remove noise and misinformation. Then, using natural language processing (NLP) techniques, the context and meaning of the user's writing are analyzed, and important decisions and thought processes are extracted. Based on this information, the server generates a personality model and thought patterns of the user. This allows for a systematic understanding of what kind of decisions the user makes in what situations.

[1513] 3. Avatar Generation Phase

[1514] The server designs an avatar based on a personality model and thought patterns it generates. This designed avatar actively utilizes the user's past data and functions as a representation of the user. Next, the terminal verifies the avatar's operation. For example, it might ask a simulated question such as, "Please explain the new market strategy." The terminal checks whether the avatar's response meets the user's expectations, and the server makes adjustments if necessary.

[1515] 4. Avatar Usage Phase

[1516] When a user is away on a business trip or busy, the avatar performs specific tasks on their behalf. For example, the avatar can give a presentation at a meeting as a task set by the user. The terminal monitors the avatar's actions and responses in real time and sends the execution log to the server. The server analyzes this log and evaluates the avatar's performance. Based on this evaluation, feedback from the user is collected, and the server improves the avatar's performance. This makes it possible to continuously utilize the user's knowledge and experience even when the user is absent or after their death. Furthermore, the avatar's performance continuously improves based on feedback, always reflecting the latest knowledge and experience.

[1517] Hardware and software to be used

[1518] Devices: PC, smartphone, tablet

[1519] Servers: High-performance server clusters (e.g., AWS EC2, Google Cloud Platform)

[1520] Software technologies: Natural language processing (e.g., Spacy, BERT), database management systems (e.g., MySQL, PostgreSQL)

[1521] Examples of specific cases and prompt statements

[1522] As a concrete example, consider a scenario where a company leader is on a long business trip, and an avatar attends internal meetings on their behalf, making decisions based on the leader's thought process and decision-making criteria. In this case, the leader can assign the following tasks to the avatar from their business trip location.

[1523] Examples of prompts to input into a generative AI model

[1524] Now, please give a presentation on our new market strategy at today's meeting. Please base your presentation on the following points:

[1525] 1. The importance and potential benefits of new markets

[1526] 2. Our strengths based on competitive analysis

[1527] 3. Specific action plan and timeline

[1528] In response to this prompt, the avatar can deliver a natural presentation based on the leader's past data and thought patterns generated by the system. The terminal monitors the avatar's actions in real time and sends feedback to the server to improve the avatar's performance.

[1529] With the above configuration, the present invention provides a system that can be widely applied by digitizing an individual's knowledge and experience and reproducing it as an avatar.

[1530] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1531] Step 1: User Data Collection Phase

[1532] Specific actions:

[1533] The device asks the user initial questions. For example, it might ask, "Please tell me about your work experience."

[1534] Input: User's response (e.g., "I have 5 years of experience as a marketing manager")

[1535] Data processing: The server analyzes this response in real time and extracts important keywords and phrases.

[1536] Output: Analyzed data (e.g., "5 years", "Marketing Manager")

[1537] Specific actions:

[1538] The server saves the analysis data to the database.

[1539] Input: Analyzed data

[1540] Output: Structured data stored in the database

[1541] Specific actions:

[1542] The server generates additional questions based on the analyzed data and sends them to the terminal.

[1543] Input: Analysis data

[1544] Data processing: Generating additional questions (e.g., "As a marketing manager, what projects did you lead?")

[1545] Output: Additional questions

[1546] Step 2: Data Analysis and Processing Phase

[1547] Specific actions:

[1548] The server cleans the collected data, removing noise and misinformation.

[1549] Input: Collected data

[1550] Data processing: Data cleaning

[1551] Output: Cleaned data

[1552] Specific actions:

[1553] The server uses NLP (Neuro-Linguistic Programming) technology to analyze context and meaning.

[1554] Input: Cleaned data

[1555] Data processing: Contextual analysis through natural language processing (e.g., using the BERT model)

[1556] Output: Information with context and meaning analyzed.

[1557] Specific actions:

[1558] The server generates a user personality model and thought patterns from the analysis results.

[1559] Input: Information analyzed for context and meaning

[1560] Data processing: Generation of personality models and thought patterns

[1561] Output: User personality model and thought patterns

[1562] Step 3: Avatar Generation Phase

[1563] Specific actions:

[1564] The avatar is designed based on personality models and thought patterns generated by the server.

[1565] Input: User's personality model and thought patterns

[1566] Data processing: Avatar design

[1567] Output: Designed avatar

[1568] Specific actions:

[1569] The device will perform a function check on the avatar.

[1570] Input: Designed avatar

[1571] Data calculation: Ask simulation questions and observe the avatar's responses (e.g., "Please explain the new market strategy").

[1572] Output: Avatar response data

[1573] Specific actions:

[1574] The server will adjust the avatar as needed.

[1575] Input: Avatar response data

[1576] Data processing: Confirmation and adjustment of responses

[1577] Output: Adjusted avatar

[1578] Step 4: Avatar Utilization Phase

[1579] Specific actions:

[1580] The avatar performs tasks set by the user.

[1581] Input: User's task settings (e.g., "Presentation at a meeting")

[1582] Data processing: Avatar performs tasks

[1583] Output: Task execution results

[1584] Specific actions:

[1585] The device monitors the avatar's movements and responses in real time and collects logs.

[1586] Input: Avatar movement data

[1587] Data processing: Collection of operation logs

[1588] Output: Collected execution logs

[1589] Specific actions:

[1590] The server analyzes the execution logs and performs a performance evaluation.

[1591] Input: Collected execution logs

[1592] Data processing: Performance evaluation

[1593] Output: Evaluation results and necessary improvements

[1594] Specific actions:

[1595] The server improves avatar performance based on user feedback.

[1596] Input: User feedback and evaluation results

[1597] Data processing: Improving avatar performance based on feedback.

[1598] Output: Improved performance avatar

[1599] (Application Example 1)

[1600] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1601] In recent years, there has been a growing demand for technology that digitizes an individual's knowledge and experience, and uses that data to generate avatars that function as digital representations. However, current technology lacks an effective means of identifying in real time what products and services a user is interested in and recommending appropriate products and services based on those preferences. Furthermore, there is a lack of means to quickly improve the avatar's behavior and responses based on user feedback. Solving these challenges is essential.

[1602] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1603] In this invention, the server includes means for asking multiple questions to collect an individual's knowledge, thoughts, experiences, and memories; means for collecting and analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for collecting the user's real-time movements and gaze via a smart terminal and analyzing the user's preferences and gaze information; and means for generating the user's avatar based on the analyzed information and recommending products and services in real time. This makes it possible to recommend products and services that reflect the user's preferences and gaze in real time.

[1604] "Personal knowledge, thoughts, experiences, and memories" refers to the collection of information, ways of thinking, actual activities and processes performed, and past experiences of a particular individual.

[1605] "Means of asking questions" refers to methods and techniques for presenting a variety of questions in order to obtain information from an individual.

[1606] "Means for collecting and analyzing response data in real time" refers to methods and technologies for quickly collecting individual responses to questions and immediately analyzing that data.

[1607] "Means for extracting important information from analyzed data and storing it in a structured database" refers to methods and techniques for analyzing collected data, identifying key elements, and storing them in a systematic database.

[1608] "Methods for generating personality models and thought patterns" refer to methods and techniques for modeling an individual's personality and thought patterns based on their behavior and responses.

[1609] "Methods for designing avatars" refers to methods and technologies for designing and creating avatars that function as extensions of an individual, based on generated personality models and thought patterns.

[1610] "Means of performing operational checks and making adjustments as necessary" refers to methods and techniques for simulating the movements of a designed avatar, verifying its suitability and functionality, and correcting any problems.

[1611] "Means of performing specific tasks" refers to methods or technologies that enable an avatar to perform specific tasks or activities on behalf of the user.

[1612] "Means for collecting execution logs and performing performance evaluations" refers to methods and technologies for collecting records of tasks performed by avatars and evaluating their results and efficiency.

[1613] "Methods for improving avatar performance based on feedback" refers to methods and technologies for improving the behavior and performance of avatars using feedback from users and systems.

[1614] "Means of collecting users' real-time actions and gaze via smart devices" refers to methods and technologies for instantly collecting users' actions and gaze via smart devices.

[1615] "Means for analyzing user preferences and gaze information" refers to methods and techniques for analyzing collected user preferences and gaze information and extracting useful patterns and trends from that data.

[1616] "Means of recommending products and services" refers to methods and technologies for presenting and recommending the most suitable products and services to users based on analyzed information.

[1617] This invention relates to a system that collects and analyzes the knowledge, thoughts, experiences, and memories of a specific individual and generates an avatar based on that information. This system is implemented using the following hardware and software.

[1618] Hardware and software to be used

[1619] Hardware: Smart glasses, webcam, server

[1620] Software: Python, OpenCV, NLP module, product recommender engine, avatar generation module

[1621] System Description

[1622] 1. User data collection phase

[1623] The server collects real-time user activity and gaze information via smart glasses. Specifically, it captures video data from the smart glasses using OpenCV and performs analysis to identify the user's gaze and interests.

[1624] 2. Data Analysis Phase

[1625] The server analyzes the collected data using NLP modules and other analytical tools. This identifies products that users are interested in and their preferences. It also performs noise reduction and misinformation cleaning to extract important data.

[1626] 3. Avatar Generation Phase

[1627] Based on the analyzed data, a user personality model and thought patterns are generated, and an avatar is created based on these. Using the avatar generation module, the avatar, which functions as a digital representation of the user, is designed in detail. This designed avatar undergoes functional verification through simulated questions based on the user's gaze and preferences.

[1628] 4. Product Recommendation Phase

[1629] The avatar provides real-time product recommendations. As the user moves around the store wearing smart glasses, the avatar uses its recommendation engine to suggest appropriate products based on their gaze. For example, if the user looks towards the electronics section, the avatar will recommend the most suitable electronic device.

[1630] 5. Feedback Phase

[1631] We receive feedback from users about the avatar's movements and responses, and improve the avatar's performance based on that feedback. We analyze user comments and activity logs and make necessary improvements.

[1632] Specific examples of processing steps

[1633] 1. The user puts on smart glasses and enters the store.

[1634] 2. The server obtains real-time video from the smart glasses and analyzes the user's gaze using OpenCV.

[1635] 3. When a user fixates their gaze on a specific product, an NLP module analyzes their preferences and interests based on that information.

[1636] 4. An avatar is generated, and a recommender engine is used to recommend products based on the analysis results.

[1637] 5. Users provide feedback on recommended products, and the server uses that information to improve the avatar's behavior.

[1638] Example of a prompt

[1639] "Please provide the following data to the NLP Processor to analyze user preferences: {user data}"

[1640] "Provide the Recommender engine with a {product list} and recommend the best products for the user."

[1641] In this way, it becomes possible to realize product recommendations that reflect the user's real-time behavior and gaze information, and to provide advanced personalized services using avatars.

[1642] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1643] Step 1:

[1644] The user wears smart glasses, and the server acquires real-time video from the smart glasses. The input is video data from the smart glasses, and the output is video frames that are processed in real time. The server uses this frame data to detect the user's gaze and analyze the gaze information.

[1645] Step 2:

[1646] The server analyzes the acquired video data using OpenCV to identify the user's gaze information. The input is real-time video frames, and the output is the user's gaze coordinate data. Based on this gaze coordinate data, the server detects objects and products that the user may be interested in.

[1647] Step 3:

[1648] The server collects basic information related to the detected products and objects. The input is a list of products detected based on gaze coordinate data, and the output is basic information related to these products (e.g., product name, category, price, etc.). The server retrieves this information and proceeds to the next analysis stage.

[1649] Step 4:

[1650] The server uses an NLP module to analyze user response data and past purchase history to identify user preferences and interests. The input is user response data and purchase history data, and the output is profile data indicating the user's preferences and interests. Based on this profile data, the server narrows down the list of product recommendation candidates.

[1651] Step 5:

[1652] The server uses a recommender engine to suggest appropriate products based on the user's profile data and gaze information. The input is profile data and gaze coordinate data, and the output is a list of recommended products. The server generates the list of recommended products, and the avatar presents it to the user.

[1653] Step 6:

[1654] The avatar presents the user with a list of recommended products and provides detailed explanations and advice. The input is the list of recommended products, and the output is a product recommendation message to the user. The server delivers this message to the user in real time through the avatar.

[1655] Step 7:

[1656] The system provides feedback on recommended products from the user. The input is user feedback data, and the output is feedback evaluation data as an analysis result. The server collects this feedback data and uses it to inform the avatar's next actions.

[1657] Step 8:

[1658] The server uses feedback evaluation data to make improvements to enhance the avatar's performance. The input is feedback evaluation data, and the output is the improved avatar behavior model. The server applies these improvements to the avatar to prepare for the next interaction.

[1659] This will allow users to receive product recommendations in real time and enjoy personalized services through avatars.

[1660] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1661] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and makes it function as a digital representation of the user. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, more natural dialogue and advanced task handling become possible. This system is implemented through the following basic phases.

[1662] 1. User data collection phase

[1663] First, the device presents the user with an initial question. For example, it might display a question such as, "Please tell me about your work experience." The user then enters their answer into the device. For example, they might enter an answer such as, "I have 5 years of experience as a marketing manager."

[1664] The server analyzes user response data in real time and extracts important keywords and phrases. For example, information such as "5 years" and "marketing manager" may be extracted. This data is stored in a structured database.

[1665] Next, the server generates additional questions based on the important information and sends them to the terminal. The emotion engine also operates simultaneously, recognizing the user's emotional state. For example, the emotion engine analyzes the user's tone of voice and facial expressions while they are answering to determine whether the user is relaxed or tense.

[1666] 2. Data Analysis and Processing Phase

[1667] The collected data is cleaned by the server to remove noise and misinformation. Next, natural language processing (NLP) techniques are used to analyze the context and meaning of the user's writing and extract important decisions and thought processes. This analysis also incorporates data from an emotion engine, taking the user's emotional state into consideration.

[1668] Based on the extracted information, the server generates a personality model and thought patterns of the user. For example, it identifies situations in which the user is likely to feel stressed and situations in which they can make calm judgments.

[1669] 3. Avatar Generation Phase

[1670] The avatar is designed based on personality models and thought patterns generated by the server. Data provided by the emotion engine is also incorporated into this process. The terminal then verifies the avatar's behavior. For example, as a simulation, it might ask the question, "Please explain the new market strategy," and verify whether the avatar responds appropriately.

[1671] In this process, the emotion engine monitors the user's emotional state based on the avatar's responses and makes necessary adjustments. For example, if the user answers a question in a relaxed tone, the avatar will be adjusted to respond in a similar tone.

[1672] 4. Avatar Usage Phase

[1673] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar can give a presentation at a meeting. The server monitors the avatar's actions and responses in real time and collects execution logs.

[1674] The server analyzes the collected logs to evaluate the avatar's performance. This evaluation also includes user emotion data provided by the emotion engine. For example, it checks whether the avatar was nervous during the presentation.

[1675] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance. For example, it might make adjustments to maintain a more relaxed tone in the next presentation.

[1676] 5. Utilizing the Emotional Engine

[1677] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar is set to respond in a calmer tone. This makes communication with the user more natural and effective.

[1678] As a concrete example, when a company leader is on a long business trip, an avatar can attend internal meetings on their behalf and make decisions based on the leader's thought process and decision-making criteria. In this scenario, an emotional engine detects any tension or stress the leader might feel during the meeting, and the avatar responds calmly to address these issues, ensuring the meeting proceeds smoothly.

[1679] As described above, the present invention provides a system that digitizes an individual's knowledge and experience and combines it with an emotion engine to reproduce it as an avatar. This makes it possible to continuously utilize the user's knowledge and experience even when the user is absent or after death, and the emotion engine makes communication more natural and effective.

[1680] The following describes the processing flow.

[1681] Step 1:

[1682] The device presents the user with an initial question. For example, it might display a question such as, "Please tell us about your work experience."

[1683] Step 2:

[1684] The user enters their answer into the device. For example, they might enter, "I have 5 years of experience as a marketing manager."

[1685] Step 3:

[1686] The server analyzes user response data in real time and extracts important keywords and phrases. For example, it might extract "5 years" or "marketing manager."

[1687] Step 4:

[1688] The device uses an emotion engine to recognize the user's emotions in real time. For example, it analyzes the user's voice tone and facial expressions to determine whether the user is relaxed or stressed.

[1689] Step 5:

[1690] The server generates additional questions based on the analyzed data and the sentiment engine data, and sends them to the terminal. For example, it might ask, "As a marketing manager, what projects have you led?"

[1691] Step 6:

[1692] The user enters their answer to an additional question on the device. For example, they might answer, "I led the marketing campaign for a new product."

[1693] Step 7:

[1694] The server also analyzes the additional response data, extracts important information, and saves it to the database. It also performs cleaning processes to remove noise and misinformation.

[1695] Step 8:

[1696] The server uses natural language processing (NLP) techniques to analyze data, understand context and meaning, and extract important decisions and thought processes. This analysis also incorporates data from the emotion engine.

[1697] Step 9:

[1698] The server generates a personality model and thought patterns of the user based on the extracted information. This takes into account the user's emotional state. For example, it identifies situations in which the user is likely to feel stressed and situations in which they are able to make calm judgments.

[1699] Step 10:

[1700] The avatar is designed based on personality models and thought patterns generated by the server. This design also incorporates data from the emotion engine.

[1701] Step 11:

[1702] The device asks the avatar simulation questions to test its functionality. For example, it might ask, "Please explain your new market strategy."

[1703] Step 12:

[1704] The device checks the avatar's response and verifies whether that response meets the user's expectations. The server adjusts the avatar's behavior as needed.

[1705] Step 13:

[1706] The emotion engine monitors the user's emotional state based on the avatar's responses and adjusts the avatar's movements and response tone accordingly. For example, it can be configured to have the avatar respond in a relaxed tone.

[1707] Step 14:

[1708] The device allows the user to finalize the avatar and collects any necessary changes or additional information. This ensures that the avatar behaves as the user expects.

[1709] Step 15:

[1710] When a user is away on a business trip or busy, the device can have an avatar perform specific tasks. For example, the avatar could give a presentation at a meeting on their behalf.

[1711] Step 16:

[1712] The server monitors the avatar's movements and responses in real time and collects execution logs.

[1713] Step 17:

[1714] The server analyzes the collected logs to evaluate the avatar's performance. This evaluation also includes user emotion data provided by the emotion engine.

[1715] Step 18:

[1716] Users provide feedback on their avatar's performance through their devices. The server uses this feedback to improve the avatar's performance. For example, it might make adjustments to maintain a more relaxed tone in the next presentation.

[1717] Step 19:

[1718] The server periodically collects new information from users and updates avatar data to the latest version. This ensures that avatars always reflect the most up-to-date knowledge and experience.

[1719] Through the steps described above, the system of the present invention reproduces the user's knowledge and experience as an avatar, and by combining it with an emotion engine, it becomes possible to have natural conversations and task responses that take into account the user's emotional state.

[1720] (Example 2)

[1721] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1722] Conventional avatar systems that utilize personal information require not only effective reproduction of the user's knowledge and experience, but also natural dialogue and task execution that takes into account the user's emotional state. However, current technology is insufficient for emotion analysis, making it difficult to adjust responses and actions in response to fluctuations in the user's emotions. Furthermore, performance evaluation of tasks performed by avatars and improvements based on feedback are not effectively carried out, thus failing to improve the quality of the user experience. In addition, cleaning processes and the use of natural language processing technologies are necessary to maintain the quality of collected data.

[1723] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for asking a number of questions to collect an individual's knowledge, thoughts, experiences, and memories; means for analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its movements, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance; means for improving the avatar's performance based on feedback; means for analyzing emotional data and adjusting the avatar's responses and movements according to the user's emotional state; means for data cleaning to remove noise and misinformation; means for analyzing context and meaning using natural language processing technology and extracting important decisions and thought processes; and means for analyzing the collected emotional data using an emotion engine. This enables natural and effective dialogue and task execution that takes into account not only the user's knowledge and experience but also their emotional state.

[1724] "Personal knowledge, thoughts, experiences, and memories" refers to the specialized knowledge, thought patterns, specific experiences and events, and memories associated with them that an individual possesses.

[1725] "Means of asking questions" refers to devices and programs for presenting questions to users and collecting their answers.

[1726] "Means for analyzing response data in real time" refers to devices and programs that process responses obtained from users immediately and extract and analyze necessary information.

[1727] A "database" refers to a system for organizing and storing collected data.

[1728] "Means for generating personality models and thought patterns" refers to devices and programs that model an individual's personality and thought patterns from collected data.

[1729] An "avatar" refers to a character that acts as a representative of a user in a digital environment.

[1730] "Means for simulating and verifying actions" refers to devices and programs that virtually test the actions of an avatar and evaluate their suitability.

[1731] "Means for performing specific tasks" refers to devices and programs that allow an avatar to perform specific roles or tasks on behalf of the user.

[1732] "Means for collecting execution logs and performing performance evaluation" refers to a device and program that records the avatar's behavior history and evaluates its performance.

[1733] "Means of improving avatar performance based on feedback" refers to devices and programs that improve avatar functionality by incorporating evaluations and opinions from users.

[1734] "Means for analyzing emotional data" refers to devices and programs that recognize a user's emotional state and analyze that data.

[1735] "Data cleaning means" refers to devices and programs that remove noise and misinformation from collected data and improve data quality.

[1736] "Natural language processing technology" refers to the technology used to understand, process, and generate human language using computers.

[1737] "Means for analyzing collected emotional data" refers to devices and programs that process emotional data obtained from users and understand its content.

[1738] This invention is a system that digitizes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and has it function as a proxy for the user. Furthermore, it integrates an emotion engine that recognizes the user's emotions, enabling more natural dialogue and advanced task handling. This system is implemented through the following basic phases.

[1739] User data collection phase

[1740] The terminal presents the user with an initial question such as, "Please tell me about your work experience." The user enters a response into the terminal, such as, "I have 5 years of experience as a marketing manager." The server analyzes this response data in real time, extracting important keywords and phrases. This data is stored in a structured database. The server then generates additional questions and sends them to the terminal. An emotion engine also operates, analyzing the user's emotional state. For example, it analyzes the user's tone of voice and facial expressions while they are answering to determine whether they are relaxed or nervous.

[1741] Data Analysis and Processing Phase

[1742] The server cleans the collected data, removing noise and misinformation. Using natural language processing (NLP) techniques, it analyzes the context and meaning of the user's writing to extract important decisions and thought processes. This analysis also incorporates data from the emotion engine, taking the user's emotional state into account. Based on the extracted information, the server generates a personality model and thought patterns of the user.

[1743] Avatar generation phase

[1744] The avatar is designed based on personality models and thought patterns generated by the server. Data provided by the emotion engine is also reflected. The terminal verifies the avatar's behavior. For example, it performs a simulation asking, "Please explain the new market strategy," and verifies whether the avatar responds appropriately. During this process, the emotion engine monitors the user's emotional state based on the avatar's responses and makes necessary adjustments.

[1745] Avatar Usage Phase

[1746] When a user is away on a business trip or busy, their device can have an avatar perform specific tasks. For example, the avatar can give a presentation at a meeting. A server monitors the avatar's actions and responses in real time and collects execution logs. The server analyzes the collected logs and evaluates the avatar's performance. This evaluation also includes user emotion data provided by an emotion engine. For example, it checks whether the avatar was nervous during the presentation. The user provides feedback through their device, and the server uses that feedback to improve the avatar's performance.

[1747] Utilizing the Emotion Engine

[1748] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar will respond in a calmer tone. As a concrete example, when a company leader is on a long business trip, the avatar attends internal meetings on their behalf and makes decisions based on the leader's thought process and judgment criteria. In this scenario, the emotion engine detects any tension or stress the leader might be experiencing, and the avatar responds accordingly with a calmer tone.

[1749] Specific example

[1750] The following scenario is an example of a prompt message.

[1751] "This scenario illustrates a situation where a company leader is on a long-term business trip, but their avatar attends an internal meeting and makes decisions based on the leader's thinking and decision-making criteria. During the meeting, an emotion engine analyzes the leader's emotional data to detect tension and stress. Based on this, the avatar provides calm responses, ensuring the meeting runs smoothly."

[1752] Hardware and software to be used

[1753] Hardware: Devices (PCs, smartphones, tablets, etc.), servers

[1754] Software: Natural Language Processing (NLP) engine, emotion engine, database system

[1755] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1756] Step 1:

[1757] Presentation of initial questions

[1758] The device displays an initial question such as, "Please tell us about your work experience." The user enters their answer using the keyboard or touch input. Specifically, a text box appears on the device screen, and the user enters, "I have 5 years of experience as a marketing manager." This input is saved on the device as text data.

[1759] Step 2:

[1760] Collection of user responses

[1761] The device collects user responses and sends them to the server. For example, it might transfer the user's response data, "I have 5 years of experience as a marketing manager," to the server. This transmitted data becomes input data for the server's analysis system.

[1762] Step 3:

[1763] Analysis of response data and keyword extraction

[1764] The server analyzes user response data in real time and extracts important keywords and phrases. Specifically, it uses natural language processing (NLP) algorithms to extract keywords such as "5 years" and "marketing manager." The input data is the user's response text, and the extracted keywords become the output data. This extracted data is structured and stored in a database.

[1765] Step 4:

[1766] Generating and submitting additional questions

[1767] The server generates additional questions based on the extracted keywords and sends them to the terminal. For example, an additional question such as "Please tell us about any specific projects or successful campaigns you have worked on recently" might be generated. The input data is the extracted keywords, and the generated additional questions become the output data.

[1768] Step 5:

[1769] Emotional analysis

[1770] The emotion engine analyzes the user's emotional state. Specifically, it analyzes voice tone and facial expression data to determine whether the user is relaxed or tense. The input data consists of recordings of the user's voice and facial expressions, and the output data is the emotional state (e.g., relaxed, tense).

[1771] Step 6:

[1772] Data Cleaning

[1773] The server cleans the collected data, removing noise and misinformation. For example, it filters out grammatical errors and irrelevant information from the input data. The input data is the user's text responses, and the cleaned text data becomes the output data.

[1774] Step 7:

[1775] Contextual analysis using natural language processing

[1776] The server uses an NLP engine to analyze the context and meaning of the user's text. Specifically, it analyzes text data to extract contextual information such as "the user has had successful marketing experiences." The input data is cleaned text data, and the contextual information becomes the output data.

[1777] Step 8:

[1778] Extraction of important decisions and thought processes

[1779] The server extracts important user decisions and thought processes from the analysis results. For example, it identifies situations where the user feels stressed or situations where they make calm judgments. The input data is the analyzed contextual information, and the characteristics of the thought process become the output data.

[1780] Step 9:

[1781] Generating personality models and thought patterns

[1782] The server generates personality models and thought patterns based on extracted thought processes. For example, it might create a model such as "Users tend to make calm judgments." The input data is the characteristics of the thought process, and the generated personality model becomes the output data.

[1783] Step 10:

[1784] Avatar design

[1785] The server designs the avatar based on the generated personality model and thought patterns. It also incorporates data from the emotion engine. The input data consists of the personality model and emotion data, while the avatar specifications become the output data.

[1786] Step 11:

[1787] Avatar operation check

[1788] The device performs a simulation, such as asking the avatar to "explain the new market strategy," to verify its operation. For example, the device asks a question, and the system checks whether the avatar answers appropriately. The input data is the question used for the simulation, and the response results become the output data.

[1789] Step 12:

[1790] Executing a specific task

[1791] The device instructs an avatar to perform specific tasks, such as giving a presentation. For example, "The avatar explains the marketing strategy for a new product at a meeting." The input data is the task content, and the output data is the result of the execution.

[1792] Step 13:

[1793] Real-time monitoring and log collection

[1794] The server monitors the avatar's actions and responses in real time and collects execution logs. The input data consists of the avatar's actions and responses, while the execution logs are the output data.

[1795] Step 14:

[1796] Performance evaluation and feedback

[1797] The server analyzes collected logs to evaluate avatar performance. It also considers user feedback to improve avatar performance. Input data consists of execution logs and feedback, while the output data represents the improvements made.

[1798] Step 15:

[1799] Adjustment based on emotional data

[1800] The emotion engine adjusts the avatar's responses and actions based on emotional data. For example, if the user is emotionally agitated, the avatar is set to respond in a calm tone. The input data is emotional data, and the adjusted responses become the output data.

[1801] (Application Example 2)

[1802] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1803] Modern virtual stores require real-time, personalized service to meet user needs for product recommendations and interactive experiences. However, traditional systems struggle to analyze users' emotional states and respond appropriately, sometimes leading to decreased satisfaction. They also struggle to provide personalized product recommendations based on individual user needs. Therefore, there is a need for a system that improves the quality of user interaction and allows users to have an experience in virtual stores that closely resembles that of physical stores.

[1804] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1805] In this invention, the server includes means for asking multiple questions to collect an individual's knowledge, thoughts, experiences, and memories; means for collecting and analyzing the individual's answer data to the questions in real time; means for extracting important information from the analyzed data and storing it in a structured database; means for generating an individual's personality model and thought patterns based on the stored data; means for designing an avatar based on the generated personality model and thought patterns; means for simulating the avatar's movements, verifying its operation, and adjusting it as necessary; means for having the avatar perform specific tasks on behalf of the user; means for collecting the avatar's execution logs and evaluating its performance, and for improving the avatar's performance based on feedback; means for interacting with the user using a smartphone and making product suggestions in a virtual store; and means for analyzing the user's emotional state using an emotion engine and adjusting the avatar's responses. This makes it possible to provide highly personalized product suggestions and services that take into account the user's emotional state.

[1806] "Personal knowledge" refers to the specialized information and experience that individual people possess.

[1807] "Thinking" refers to the process of problem-solving and decision-making that users engage in.

[1808] "Experience" refers to the knowledge and skills that an individual has learned from their past actions and experiences.

[1809] "Memory" refers to an individual's ability to retain past events and information.

[1810] A "question" refers to a question posed to a user in order to obtain information.

[1811] "Response data" refers to the information and responses that users provide in response to questions.

[1812] "Real-time analysis" refers to the process of analyzing data immediately on the spot.

[1813] "Important information" refers to data that is deemed particularly valuable among the analyzed data.

[1814] A "structured database" refers to a database that is organized and stored according to a specific format or set of rules.

[1815] A "personality model" refers to a model that represents the personality traits of individual users.

[1816] "Thinking patterns" refer to the series of actions and decision-making tendencies that users exhibit when solving problems.

[1817] An "avatar" refers to a virtual person or character that acts on behalf of a user.

[1818] "Simulation" refers to the act of virtually recreating a real environment to verify its operation.

[1819] "Performing a specific task" refers to carrying out pre-defined tasks or duties.

[1820] An "execution log" refers to a record of the actions and responses performed by the avatar.

[1821] "Performance evaluation" refers to the process of evaluating the performance of an avatar.

[1822] A "smartphone" refers to a portable, multi-functional information terminal.

[1823] "Dialogue" refers to communication between the user and the system.

[1824] A "virtual store" refers to a virtual sales environment that exists on the internet.

[1825] "Product recommendation" refers to the act of recommending appropriate products to users.

[1826] An "emotion engine" refers to a technology that analyzes a user's emotional state and adjusts its response accordingly.

[1827] "Adjusting your response" means changing your answer to suit the other person's situation and feelings.

[1828] This invention relates to a system that provides personalized product suggestions to users in a virtual store and adjusts responses according to their emotions. Specifically, it provides a system that collects and analyzes an individual's knowledge, thoughts, experiences, and memories, generates an avatar based on that data, and has the avatar perform specific tasks on behalf of the user. The specific configuration and method for carrying out this invention are described below.

[1829] Hardware and software

[1830] This system consists of a smartphone, a server, a natural language processing engine (NLP), an emotion engine, a database, and a front-end application.

[1831] Hardware: Smartphone (iOS or Android device)

[1832] software:

[1833] Natural Language Processing Engine (NLP): Uses the Google NLP API.

[1834] Emotion engine: Uses Microsoft Azure Emotion API

[1835] Database: Firebase Realtime Database is used.

[1836] Frontend application: Developed using React Native.

[1837] Data collection and analysis

[1838] In the initial stages of the system, the server collects data about the user's knowledge, thoughts, experiences, and memories. This involves a process where the user is asked multiple questions via their smartphone and provides answers. This response data is analyzed in real time by the server, and important information is extracted.

[1839] Data storage and modeling

[1840] The analyzed data is filtered to remove noise and misinformation before being stored in a structured format in the Firebase Realtime Database. Then, natural language processing (NLP) techniques are used to analyze the context and meaning, generating a user personality model and thought patterns.

[1841] Avatar generation and simulation

[1842] The avatar is designed based on personality models and thought patterns generated by the server. The avatar design also incorporates data from an emotion engine to reflect the user's emotional state. Finally, the avatar's movements are simulated and verified.

[1843] Using avatars

[1844] When a user uses a virtual store, an avatar performs specific tasks on their behalf, such as suggesting products. Specifically, it interacts with the user using a smartphone, analyzes the user's emotional state using an emotion engine, and adjusts its response accordingly.

[1845] Feedback and Performance Evaluation

[1846] Avatar execution logs are collected and performance is evaluated. This includes user feedback, and the collected feedback can be used to improve the avatar's performance. This feedback loop allows for continuous improvement of the avatar's behavior and responsiveness.

[1847] Examples of specific cases and prompt statements

[1848] The following interactions are examples of what can be considered.

[1849] Example of a prompt:

[1850] "Please tell me about your work experience."

[1851] User response:

[1852] "I have five years of experience as a marketing manager."

[1853] The data collected in this way is analyzed, and an avatar is generated that interacts with the user in the virtual store based on the user's personality model. This makes it possible to provide highly personalized product suggestions and services that take into account the user's emotional state.

[1854] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1855] Step 1:

[1856] The server initiates a process to collect the user's knowledge, thoughts, experiences, and memories. Specifically, it asks the user multiple questions via their device (smartphone). The user's responses to these questions are collected and analyzed in real time.

[1857] Input: Response data entered by the user on the device.

[1858] Output: Set of collected response data

[1859] Step 2:

[1860] The server analyzes the collected response data in real time and extracts important information. It uses the Google NLP API to analyze the context and meaning of the response data and extract key keywords and phrases.

[1861] Input: Collected response data

[1862] Output: Important keywords and phrases

[1863] Step 3:

[1864] The server stores the extracted key keywords and phrases in a structured database (Firebase Realtime Database). This data will be used for future analysis and avatar generation.

[1865] Input: Important keywords or phrases

[1866] Output: Structured data stored in the database

[1867] Step 4:

[1868] The server generates individual personality models and thought patterns based on stored data. Natural language processing (NLP) techniques are used to model user characteristics and patterns.

[1869] Input: Structured data stored in a database

[1870] Output: Personality model and thought patterns

[1871] Step 5:

[1872] The server designs avatars based on generated personality models and thought patterns. It also integrates data from emotion engines such as the Microsoft Azure Emotion API to consider emotion-based responses.

[1873] Input: Data from personality models, thought patterns, and emotion engines.

[1874] Output: Designed avatar

[1875] Step 6:

[1876] The server simulates and verifies the avatar's movements. This process verifies whether the avatar can respond appropriately and perform specific tasks. Adjustments are made as needed.

[1877] Input: Designed avatar

[1878] Output: Avatars whose operation has been confirmed

[1879] Step 7: ...

Claims

1. A means of asking multiple questions to collect an individual's knowledge, thoughts, experiences, and memories, A means for collecting and analyzing individual response data to the aforementioned questions in real time, A means for extracting important information from the analyzed data and storing it in a structured database, A means of generating an individual's personality model and thought patterns based on stored data, A means for designing an avatar based on the aforementioned generated personality model and thought patterns, A means to simulate the avatar's movements, verify them, and make adjustments as needed. A means of having the avatar perform a specific task on behalf of the user, A means for collecting the execution logs of the aforementioned avatar and performing performance evaluation, A means to improve avatar performance based on feedback, A system that includes this.

2. A means for cleaning the collected data from noise and misinformation, A means of analyzing the context and meaning of cleaned data using natural language processing (NLP) techniques to extract important decisions and thought processes, The system according to claim 1, including the following:

3. When the aforementioned avatar performs a specific task according to the user's wishes, means for causing the avatar to perform a specific action based on the task content set by the user, The aforementioned avatar receives feedback after performing a task according to the user's wishes, and means for improving the avatar's behavior and responses based on the feedback, The system according to claim 1, including the following:

Images