System

Abstract

A system is provided.SOLUTION: A system, comprising: means for collecting data of an individual; means for sending the collected data to a server; means for preprocessing the data received by the server and generating a user model using a machine learning algorithm; means for creating a digital clone using the generated user model and sending the digital clone to a terminal; and means for the digital clone to perform an online task according to an instruction of a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, many people are extremely busy with work and family life, making it difficult to efficiently manage tasks and complete online tasks. In these situations, many people wish they had a second self, but having a physical alter ego is impossible. Furthermore, existing task management systems and online work support tools cannot fully address users' individual needs, leaving them unable to receive sufficient support. The present invention aims to solve these problems by providing a new digital clone system that reduces users' burden and improves work efficiency. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means.

[0006] The means by which personal data is collected;

[0007] means for transmitting the collected data to a server;

[0008] a means for preprocessing the received data by the server and generating a user model using a machine learning algorithm;

[0009] A means for creating a digital clone using the generated user model and transmitting the digital clone to a terminal;

[0010] The system includes a means for a digital clone to perform online tasks according to a user's instructions.

[0011] Furthermore, the digital clone can participate in online meetings, convert speech into text in real time to create minutes, and report the created minutes to the user, or it can analyze the user's schedule, list high-priority tasks, and set reminders, thereby providing even more advanced support tailored to the user's needs.

[0012] As a result, the present invention generates a digital clone that meets individual needs based on user data, automating online task management and work assistance, reducing the burden on users and improving work efficiency.

[0013] "Personal data" refers to information about a user, including various digital information such as email history, schedule information, audio data, photos, and Internet history.

[0014] "Collection Methods" refers to the software and hardware mechanisms used to obtain and store data from users.

[0015] "Server" refers to a computer system that centrally processes and stores collected data and transmits information to terminals as needed.

[0016] "Preprocessing" refers to data manipulation operations such as data cleaning, normalization, and filtering to convert raw data into an analyzable format.

[0017] A "machine learning algorithm" is a type of computer program used for data analysis and model generation, which learns user behavior patterns and characteristics from data.

[0018] "User Model" means a mathematical and digital representation of a user's behavioral characteristics and preferences created by machine learning algorithms.

[0019] A "digital clone" refers to a software agent that is created based on a generated user model and mimics the user's role.

[0020] A "terminal" is a device that is directly operated by a user, and includes a personal computer, smartphone, tablet, etc.

[0021] "Online tasks" refers to various tasks and activities performed via the Internet, such as online meetings, email management, and schedule adjustments.

[0022] "Real-time text conversion" refers to the process of instantly converting speech data into text data.

[0023] "Minutes" refers to a document that records what was said and what decisions were made during a conference or meeting.

[0024] A "reminder" is a message or alert that notifies a user of a specific task or schedule.

[0025] "Analysis" refers to the process of examining data or information in detail to clarify its structure, content, and relationships.

[0026] By defining each term in this manner, the scope and content of the present invention will become clearer. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0028] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0029] First, the terms used in the following description will be explained.

[0030] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0031] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0032] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0033] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0034] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0035] [First embodiment]

[0036] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0037] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0038] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0039] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0040] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0041] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0042] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0043] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0044] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0045] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0046] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0047] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] The present invention relates to a system for creating a digital clone using personal data of a user and assisting with online tasks. Specific embodiments of the system are described below.

[0049] Digital Clone System Overview

[0050] The system of the present invention is composed of three main components: a "server," a "terminal," and a "user." Users interact with the system using dedicated applications. Data collection, processing, cloning, and task assistance are performed, and each step to achieve this is described in detail below.

[0051] User Data Collection

[0052] Users provide their own data, including email history, schedules, voice data, photos, internet history, etc. Users select this data through a dedicated application and upload it to the system.

[0053] The device temporarily stores this data and transmits it to a server in encrypted form using a secure protocol.

[0054] Data Preprocessing and Training

[0055] The server preprocesses the received data, which includes cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences and generate a user model. This model serves as the basis for creating a digital clone of the user.

[0056] Creating a digital clone

[0057] The server generates a digital clone based on the learned user model. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions.

[0058] The device stores the received digital clone locally and activates it when the user wants to use it.

[0059] Online task assistance

[0060] The user instructs the digital clone to perform online tasks, such as "attend tomorrow's online meeting and take minutes."

[0061] The device analyzes this instruction and requests the necessary information from the server, which then collects meeting information from the user's schedule management system and sends the details to the device.

[0062] The digital clone automatically logs in at the designated meeting time, performs real-time voice recognition, and converts the conversation into text. When the meeting ends, a summary of the meeting minutes is sent to the user via the device.

[0063] Specific examples

[0064] Example 1: Creating minutes for online meetings

[0065] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0066] 2. The device sends this instruction to the server.

[0067] 3. The server collects the meeting details (participant list, agenda, etc.) and sends them to the terminal.

[0068] 4. A digital clone logs in at the meeting time and recognizes and transcribes the conversation in real time.

[0069] 5. The digital clone compiles minutes and sends them to the user via the terminal after the meeting ends.

[0070] Example 2: Task Management

[0071] 1. The user asks, "Organize this week's tasks and set reminders."

[0072] 2. The device sends the user's schedule to the server.

[0073] 3. The server analyzes the user's schedule and identifies high-priority tasks.

[0074] 4. The server lists the identified tasks, sets reminders and sends them to the device.

[0075] 5. The device notifies the user of the reminder.

[0076] How to use

[0077] To use the system of the present invention, a user installs a dedicated application and registers as a user. Next, the user selects and provides the data they want to collect, and begins creating a digital clone. Once the digital clone is created, the user can instruct the digital clone to perform various online tasks through the dedicated application, improving work efficiency.

[0078] In this way, by implementing the digital clone system of the present invention, users can efficiently manage and carry out their own work, thereby reducing their workload.

[0079] The processing flow will be explained below.

[0080] Digital clone system processing flow

[0081] Data Collection Phase

[0082] Step 1:

[0083] The user launches the dedicated application and selects the data they want to collect (emails, schedules, audio data, photos, internet history, etc.).

[0084] Step 2:

[0085] The device temporarily stores selected data, for example, email history in a temporary folder in local storage.

[0086] Step 3:

[0087] The device encrypts the stored data using a secure communication protocol (e.g., SSL / TLS) and sends it to the server.

[0088] Data preprocessing and training phase

[0089] Step 4:

[0090] The server preprocesses the data it receives. Specifically, it cleans and filters the data to remove unnecessary noise. For example, it removes spam and advertising emails from email data.

[0091] Step 5:

[0092] The server feeds the preprocessed data into machine learning algorithms to generate a user model, a process that learns and models the user's behavioral patterns and preferences.

[0093] Digital Clone Generation Phase

[0094] Step 6:

[0095] The server generates a digital clone based on the learned user model. This digital clone is an agent that simulates the user's actions and decisions.

[0096] Step 7:

[0097] The server converts the generated digital clone into binary format and transmits it to the terminal.

[0098] Step 8:

[0099] The device stores the received digital clone locally and launches it when the user needs it.

[0100] Online task assistance phase

[0101] Step 9:

[0102] The user gives the digital clone tasks through a dedicated application, such as "attend tomorrow's online meeting and take minutes."

[0103] Step 10:

[0104] The terminal analyzes the user's instructions and sends a request to the server to obtain the required information.

[0105] Step 11:

[0106] The server collects the user's schedule and meeting information and sends it to the device, for example extracting meeting details from the user's calendar.

[0107] Step 12:

[0108] Based on the information acquired by the device, a digital clone is activated and allowed to participate in the meeting. The digital clone converts speech into text in real time and creates minutes.

[0109] Step 13:

[0110] After the meeting, the digital clone compiles minutes and sends them to the user via the device.

[0111] Step 14:

[0112] The user reviews the minutes they receive and gives the digital clone further instructions, such as "Share the minutes with the team."

[0113] Specific examples

[0114] Example 1: Process flow for creating minutes of an online meeting

[0115] Step 1:

[0116] The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0117] Step 2:

[0118] The terminal analyzes the instructions and sends them to the server.

[0119] Step 3:

[0120] The server collects the details of the meeting (participant list, agenda, etc.) and sends them to the terminal.

[0121] Step 4:

[0122] The digital clone automatically logs in at meeting time and recognizes voice and transcribes the conversation in real time.

[0123] Step 5:

[0124] The digital clone compiles minutes and sends them to the user via the terminal after the meeting ends.

[0125] Example 2: Task management process flow

[0126] Step 1:

[0127] The user says, "Organize this week's tasks and set reminders."

[0128] Step 2:

[0129] The terminal transmits the user's schedule information to the server.

[0130] Step 3:

[0131] The server analyzes the schedule and identifies important and urgent tasks.

[0132] Step 4:

[0133] The server lists the identified tasks and sends them to the terminal.

[0134] Step 5:

[0135] The device notifies the user of the reminder.

[0136] These detailed processing steps allow the digital clone system of the present invention to efficiently perform tasks according to the user's instructions, thereby reducing the burden on the user.

[0137] Example 1

[0138] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] In today's world, with the increase in online tasks, it is becoming increasingly difficult for individual users to efficiently manage their time and complete tasks. Furthermore, taking minutes of online meetings and managing tasks is time-consuming and accuracy is an issue. This has led users to seek ways to improve the efficiency of their work.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0141] In this invention, the server includes means for collecting personal data, means for transmitting the collected data to the server using an encryption protocol, means for the server to preprocess the received data and generate a user model using a machine learning algorithm, means for the server to create a digital clone using the generated user model and transmit it to the terminal, means for the digital clone to analyze the user's instructions using natural language processing and execute online tasks, and means for the digital clone to notify the user of the results after the task is executed. This enables users to efficiently manage and execute online tasks, significantly improving work efficiency.

[0142] "Personal data" refers to any form of information related to a user (email history, schedule, audio data, photos, internet history, etc.).

[0143] "Cryptographic protocol" refers to the technology (e.g., TLS) used to ensure the secure communication of data.

[0144] A "server" refers to a computer system that performs core computations, such as data preprocessing, generating machine learning models, and creating digital clones.

[0145] A "machine learning algorithm" refers to a computational method that learns patterns from data and generates and updates a user model.

[0146] A "user model" refers to a data structure that represents a user's behavioral patterns and preferences, generated by a machine learning algorithm based on collected personal data.

[0147] A "digital clone" refers to a software agent that is generated based on a user model and mimics the user's behavior and decisions.

[0148] "Terminal" refers to the device (e.g., smartphone, tablet, PC) through which a user interacts with their digital clone.

[0149] "Natural language processing" refers to the technology that a digital clone uses to understand and interpret user instructions.

[0150] "Online tasks" refer to Internet tasks that a user instructs a digital clone to perform (e.g., participating in an online meeting, task management).

[0151] "Speech recognition technology" refers to technology that converts speech into text in real time.

[0152] "Minutes" refers to a document that records the contents of the conversation that took place during an online meeting as text.

[0153] "Reminder" refers to a function that notifies the user according to a schedule.

[0154] "Information collection system" refers to external APIs and tools for collecting user schedules and other necessary information.

[0155] The present invention relates to a system for creating a digital clone using personal data of a user and assisting with online tasks. Specific embodiments of the system are described below.

[0156] User Data Collection

[0157] Users use a dedicated application to provide their own data, including email history, schedules, voice data, photos, internet history, etc. The user selects this data and uploads it to the system.

[0158] The device temporarily stores this data and transmits it to the server using an encryption protocol such as TLS (Transport Layer Security).

[0159] Data Preprocessing and Training

[0160] The server preprocesses the received data, including cleaning the data (e.g., imputing missing values ​​and removing outliers), filtering, and normalization. This preprocessing is performed using Python's Pandas and Numpy libraries.

[0161] The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences, generating a user model using libraries like TensorFlow and PyTorch. This user model serves as the basis for creating a digital clone of the user.

[0162] Creating a digital clone

[0163] The server generates a digital clone based on the learned user model. This digital clone is a software agent generated using Python scripts that mimics the user's behavior and decisions. The generated digital clone is converted into a binary format and sent to the device.

[0164] The terminal stores the received digital clone in local storage, using SQLite or JSON format. When the user wants to use it, the terminal launches the digital clone, possibly using a Python script or a Docker container.

[0165] Online task assistance

[0166] The user instructs the digital clone to perform online tasks. These instructions are entered as prompt sentences through the interface of a dedicated application. For example, the user might instruct the clone to "attend tomorrow's online meeting and take minutes."

[0167] The device parses this instruction using natural language processing (NLP) techniques (e.g., spaCy or NLTK libraries). The device then requests the necessary information from the server. For example, it could retrieve meeting information from the Google Calendar API.

[0168] The server collects meeting information from the user's scheduler and sends the details to the device. The digital clone then automatically logs in at the designated meeting time and performs real-time speech recognition and converts the conversation into text, using, for example, the Google Cloud Speech-to-Text API. Once the meeting is over, a summary of the meeting minutes is reported to the user via the device.

[0169] Specific examples

[0170] Example 1: Creating minutes for online meetings

[0171] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0172] 2. The device analyzes this instruction and requests the necessary information from the server.

[0173] 3. The server uses the Google Calendar API to collect the meeting details and send them to the device.

[0174] 4. At the meeting time, the digital clone logs in using Selenium and converts the audio to text using the Google Cloud Speech-to-Text API.

[0175] 5. The digital clone compiles the minutes in Markdown format and notifies the user through a dedicated application.

[0176] Example 2: Task Management

[0177] 1. The user asks, "Organize this week's tasks and set reminders."

[0178] 2. The terminal sends the user's schedule information to the server.

[0179] 3. The server uses Natural Language Processing (NLP) technology to analyze the schedule and identify high-priority tasks.

[0180] 4. The server lists the identified tasks, sets reminders using the Google Calendar API, and sends them to the device.

[0181] 5. The device notifies the user of the reminder.

[0182] Prompt Sentence Examples

[0183] "Please take the minutes of the meeting tomorrow at 9 o'clock."

[0184] "Organize my tasks for this week and set reminders."

[0185] By following this specific process flow, users can efficiently manage and complete online tasks.

[0186] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0187] Step 1:

[0188] The user launches a dedicated application and provides their personal data (e.g., email history, schedule, audio data, photos, internet history).

[0189] Input: Personal Data

[0190] Output: Temporarily saved dataset

[0191] Specific operations: Select the type of data to provide through the dedicated application interface and press the button to upload the data.

[0192] Step 2:

[0193] The device temporarily stores the data uploaded by the user and sends it to the server using an encryption protocol such as TLS (Transport Layer Security).

[0194] Input: Temporarily saved dataset

[0195] Output: Data in encrypted form

[0196] Specific operation: The data is temporarily stored in a local database (e.g., SQLite) on the device, and then encrypted using the OpenSSL library. After encryption, the data is sent to the server.

[0197] Step 3:

[0198] The server stores the received data in a cloud database (e.g., Amazon RDS) and performs preprocessing.

[0199] Input: Encrypted data

[0200] Output: Preprocessed dataset

[0201] Specific operations: Decrypt the data and use the Pandas library to clean (fill in missing values, remove outliers), filter, and normalize the data.

[0202] Step 4:

[0203] The server inputs the preprocessed data into a machine learning algorithm (e.g., TensorFlow, PyTorch) to generate a user model.

[0204] Input: Preprocessed dataset

[0205] Output: User model

[0206] Specific operation: Data is fed into a machine learning algorithm to learn behavioral patterns and preferences. As a result, a user model is generated.

[0207] Step 5:

[0208] The server generates a digital clone based on the user model, converts it into binary format, and transmits it to the terminal.

[0209] Input: User model

[0210] Output: Digital clone in binary format

[0211] Specific operation: The user model is embodied as a digital clone using a Python script, then converted into binary format. The converted digital clone is sent to the terminal.

[0212] Step 6:

[0213] The terminal stores the received digital clone in local storage (e.g., SQLite, JSON format) and activates the digital clone as needed.

[0214] Input: Digital clone in binary format

[0215] Output: Activated digital clone

[0216] Specific operation: A digital clone is stored in local storage and launched using a Python script or Docker container when the user uses it.

[0217] Step 7:

[0218] The user instructs the digital clone on tasks through a dedicated application.

[0219] Input: Prompt sentence (e.g., "Please attend tomorrow's online meeting and take minutes.")

[0220] Output: Parsed task instructions

[0221] Specific operations: Enter the prompt text through the dedicated application interface and press the send button.

[0222] Step 8:

[0223] The device analyzes the user's instructions using natural language processing (NLP) technology (e.g., spaCy, NLTK) and requests the necessary information from the server.

[0224] Input: prompt statement

[0225] Output: Task execution information request

[0226] Specific operations: Analyzes the text of the user's instructions using NLP technology, interprets the task content, and sends a request to the server for the appropriate information.

[0227] Step 9:

[0228] The server collects the necessary information from the user's schedule management system (e.g., Google Calendar API) and sends it to the device.

[0229] Input: Task execution information request

[0230] Output: Collected schedule information

[0231] Specific operation: Uses API to obtain user schedule information and sends it to the device.

[0232] Step 10:

[0233] The digital clone performs online tasks (e.g., participating in an online meeting) and notifies the user of the results.

[0234] Input: Collected schedule information

[0235] Output: Task results (e.g. meeting minutes)

[0236] Specific operations: Log in at the specified meeting time (e.g., using Selenium), convert the audio into text using the Google Cloud Speech-to-Text API, summarize the task results, and notify the user.

[0237] Prompt Sentence Examples

[0238] "Please take the minutes of the meeting tomorrow at 9 o'clock."

[0239] "Organize my tasks for this week and set reminders."

[0240] The above is the specific processing flow of this system.

[0241] (Application example 1)

[0242] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0243] While existing technologies exist that generate digital clones based on a user's personal data and assist with online tasks, they lack the means to analyze a customer's purchase history and preferences in physical stores and recommend appropriate products. As a result, improvements to customer service and the shopping experience in physical stores have not been fully realized. The present invention aims to solve this problem and improve the customer experience in physical stores.

[0244] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0245] In this invention, the server includes a means for acquiring personal data, a means for transmitting the acquired data to the server, a means for the server to preprocess the received data and generate a user model using a machine learning algorithm, a means for creating a digital clone using the generated user model and transmitting it to the terminal, a means for the digital clone to perform online tasks according to the user's instructions, and a means for the digital clone to analyze the customer's purchase history and preferences using smart glasses in a physical store and recommend appropriate products. This enables the digital clone to immediately respond to customer needs even in a physical store and recommend optimal products.

[0246] "Personal data" refers to data that includes a user's behavioral history, purchasing history, preferences, and other personal information.

[0247] A "server" is a computer system that receives user data, pre-processes it, generates models using machine learning, and creates and transmits digital clones.

[0248] A "terminal" is a device that receives a digital clone sent from a server and allows a user to interact with the digital clone.

[0249] A "machine learning algorithm" is a technology that analyzes user data and learns user behavior patterns and preferences.

[0250] A "user model" is a data model that reflects a user's behavioral patterns and preferences, generated using a machine learning algorithm.

[0251] A "digital clone" is a software agent that mimics a user's behavior and decisions based on a generated user model.

[0252] An "online task" is a task performed over the Internet and executed by the digital clone based on the user's instructions.

[0253] A "physical store" is a physically existing sales store that is a place that users (customers) visit in person.

[0254] "Smart glasses" are wearable devices that have augmented reality and information display functions and are used by store clerks in physical stores when providing customer service.

[0255] "Purchase history" is a record of products that a customer has purchased in the past.

[0256] "Preferences" refer to a user's tendency to prefer a particular type or style of product.

[0257] In the context of "Levidence," a "designated document" is a document submitted in the course of a trial or investigation that is referenced based on specific requirements.

[0258] The present invention is a system for collecting personal data of users and generating digital clones, and it is particularly intended to provide personalized product recommendations to customers using smart glasses in brick-and-mortar stores. Specific embodiments of the system are described below.

[0259] System Overview

[0260] The system of the present invention consists of a server, a terminal, and a user. The user interacts with the system using smart glasses, and data collection, processing, digital cloning, and product recommendations are performed. This procedure is described in detail below.

[0261] Retrieving User Data

[0262] Users provide their own data, including purchase history, preferences, and other behavioral patterns. They select this data through a dedicated application and upload it to the system. The device temporarily stores this data and transmits it to the server in encrypted form using a secure protocol.

[0263] Data preprocessing and user model generation

[0264] The server preprocesses the received data, which includes cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms (e.g., the scikit-learn library) to learn the user's behavioral patterns and preferences and generate a user model.

[0265] Creating and transmitting digital clones

[0266] The server generates a digital clone based on the learned user model. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions. The device stores the received digital clone locally and activates it when the user uses it.

[0267] Product recommendations in physical stores

[0268] A user wears smart glasses and performs the task of recommending products to a customer. For example, a customer may request, "Please show me the latest new products." The device sends this instruction to the server, which analyzes the customer's purchase history and preferences. Based on the analysis results, the device sends a recommendation list of appropriate new products. The user can then suggest the best products to the customer based on the information displayed on the smart glasses' display.

[0269] Specific examples

[0270] Usage Scenarios

[0271] 1. Customer A asks the store clerk to show him the latest new products.

[0272] 2. The device sends this instruction to the server.

[0273] 3. The server analyzes customer A's purchase history and preferences, generates a list of recommended products, and sends it to the terminal.

[0274] 4. The user (store clerk) introduces the recommended product to customer A through the smart glasses.

[0275] Prompt Sentence Examples

[0276] "Customer ID: customer123. Looking at their recent purchase history, they have purchased a bag, shoes, and a jacket. New items have arrived. What products would you recommend for this customer?"

[0277] Hardware and software used

[0278] Hardware: Servers (e.g., Amazon Web Services, Google Cloud Platform)

[0279] Software: Python libraries (e.g., requests, scikit-learn)

[0280] This will enable the digital clone to instantly respond to customer needs in physical stores and recommend the most suitable products.

[0281] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0282] Step 1:

[0283] User provides data

[0284] Users select data such as their purchase history, preferences, and behavioral patterns through a dedicated application and upload that data to the device. INPUT is the user's personal information and history data, and OUTPUT is encrypted data stored on the device.

[0285] Step 2:

[0286] The device sends data to the server

[0287] The device temporarily stores the acquired data and sends it to the server in encrypted form using a secure protocol (e.g., HTTPS). INPUT is the encrypted data stored on the device, and OUTPUT is the encrypted data sent to the server.

[0288] Step 3:

[0289] The server preprocesses the data

[0290] The server performs pre-processing on the received data, such as cleaning, filtering, and normalization. Unwanted noise is removed and the data is in a format that can be analyzed. INPUT is the encrypted data sent to the server, and OUTPUT is the pre-processed data.

[0291] Step 4:

[0292] The server generates a user model using a machine learning algorithm

[0293] The server uses the preprocessed data to apply machine learning algorithms (e.g., scikit-learn) to learn the user's behavioral patterns and preferences, which results in a user model. The input is the preprocessed data, and the output is the user model.

[0294] Step 5:

[0295] The server uses the user model to create a digital clone

[0296] The server creates a digital clone based on the generated user model. The created digital clone is converted into binary format and sent to the terminal. The input is the user model, and the output is the binary format digital clone.

[0297] Step 6:

[0298] The device stores the digital clone locally and activates it when the user wants to use it.

[0299] The terminal stores the received digital clone locally and activates it as needed. This allows the user to receive services from the digital clone. INPUT is the digital clone sent from the server, and OUTPUT is the digital clone stored in the terminal.

[0300] Step 7:

[0301] Users wear smart glasses to serve customers

[0302] The user (store clerk) wears smart glasses and receives requests from customers (e.g., "Please show me the latest new products"). INPUT is the customer's request, and OUTPUT is the instruction sent to the terminal.

[0303] Step 8:

[0304] The terminal sends the customer's request to the server

[0305] The terminal analyzes the customer's request and sends the information to the server. INPUT is the customer's request, and OUTPUT is the request to the server.

[0306] Step 9:

[0307] The server analyzes the customer's purchase history and preferences to recommend appropriate products.

[0308] The server analyzes the customer's purchase history and preferences based on the received request. This generates an appropriate product recommendation list and sends it to the terminal. INPUT is the request to the server and customer data, and OUTPUT is the list of recommended products.

[0309] Step 10:

[0310] Users can suggest products to customers through smart glasses

[0311] The user (store clerk) recommends the best products to the customer based on the product recommendation list displayed on the smart glasses display. The input is the list of recommended products, and the output is the product proposal to the customer.

[0312] This allows the digital clone to instantly respond to customer needs in a physical store and recommend the most suitable products.

[0313] "Customer ID: customer123. Looking at their recent purchase history, they have purchased a bag, shoes, and a jacket. New items have arrived. What products would you recommend for this customer?"

[0314] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0315] The present invention relates to a system for creating a digital clone using personal data and emotional data of a user and assisting the user in online tasks. Specific embodiments of the system are described below.

[0316] Digital Clone System Overview

[0317] The system of the present invention consists of four main components: a "server," a "terminal," a "user," and an "emotion engine." The user interacts with the system using a dedicated application, which collects data, processes it, creates clones, and assists with tasks. The emotion engine also recognizes the user's emotions in real time and utilizes this information throughout the system.

[0318] User and sentiment data collection

[0319] The user selects the data they want to collect (email, schedule, voice data, photos, internet history, etc.) through a dedicated application, and simultaneously collects emotional data (voice and facial expression data). Emotional data is acquired through sensors such as a microphone and camera.

[0320] The device temporarily stores this data and transmits it to a server in encrypted form using a secure protocol.

[0321] Data Preprocessing and Training

[0322] The server preprocesses the received data. This involves cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences and generate a user model. This model serves as the basis for creating a digital clone of the user.

[0323] The server similarly processes the emotion data obtained through the emotion engine and incorporates the user's emotional state into the model.

[0324] Creating a digital clone

[0325] The server generates a digital clone based on the trained user model and emotional data. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions, and responds taking into account the user's emotional state.

[0326] The device stores the received digital clone locally and activates it when the user wants to use it.

[0327] Online task assistance

[0328] The user instructs the digital clone to perform online tasks, such as "attend tomorrow's online meeting and take minutes."

[0329] The device analyzes this instruction and requests the necessary information from the server, which then collects meeting information from the user's schedule management system and sends the details to the device.

[0330] The digital clone automatically logs in at the designated meeting time, performs real-time speech recognition, and converts the conversation into text. When the meeting ends, a summary of the meeting minutes is sent to the user via their device. In addition, an emotion engine monitors the user's emotional state during the meeting and creates appropriate responses and meeting minutes based on that information.

[0331] Specific examples

[0332] Example 1: Creating minutes for online meetings

[0333] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0334] 2. The device sends this instruction to the server.

[0335] 3. The server collects the meeting details (participant list, agenda, etc.) and sends them to the terminal.

[0336] 4. The digital clone automatically logs in at the meeting time, recognizes the conversation in real time, and transcribes it into text. The emotion engine simultaneously monitors the person's emotional state.

[0337] 5. The digital clone compiles minutes of the meeting, taking into account emotional data, and sends them to the user via their device after the meeting ends.

[0338] Example 2: Task Management

[0339] 1. The user asks, "Organize this week's tasks and set reminders."

[0340] 2. The terminal sends the user's schedule information to the server.

[0341] 3. The server analyzes the schedule and identifies important and urgent tasks, taking into account emotional data.

[0342] 4. The server lists the identified tasks and sends them to the device.

[0343] 5. The device notifies the user of the reminder. The emotion engine provides feedback on the user's emotional state and adjusts the wording of the reminder.

[0344] How to use

[0345] To use the system of the present invention, a user installs a dedicated application and registers as a user. Next, the user selects and provides the data they want to collect, and begins creating a digital clone. Once the digital clone is created, the user can instruct the digital clone to perform various online tasks through the dedicated application, improving work efficiency.

[0346] Furthermore, the introduction of an emotion engine enables the digital clone to respond and act in a way that takes into account the user's emotional state, providing more appropriate and effective support.

[0347] In this way, by implementing the digital clone system of the present invention, users can efficiently manage and carry out their own work, reducing their workload. The introduction of an emotion engine will provide even more advanced support.

[0348] The processing flow will be explained below.

[0349] Digital clone system processing flow

[0350] Data Collection Phase

[0351] Step 1:

[0352] The user launches the dedicated application and selects the data they want to collect (emails, schedules, audio data, photos, internet history, etc.).

[0353] Step 2:

[0354] The device temporarily stores the selected data, while simultaneously collecting the user's voice and facial expression data using the microphone and camera.

[0355] Step 3:

[0356] The device analyzes the emotional state using voice and facial expression data and generates emotional data.

[0357] Step 4:

[0358] The personal data and emotional data collected by the device are sent to a server using a secure protocol (e.g., encryption using SSL / TLS).

[0359] Data preprocessing and training phase

[0360] Step 5:

[0361] The server preprocesses the received personal and emotional data, specifically cleaning, filtering, and denoising the data.

[0362] Step 6:

[0363] The server then feeds the pre-processed data into machine learning algorithms to generate a user model that reflects the user's behavioral patterns and emotional state.

[0364] Digital Clone Generation Phase

[0365] Step 7:

[0366] The server generates a digital clone based on the learned user model and emotional data. The digital clone simulates the user's behavior and decisions and responds according to the user's emotions.

[0367] Step 8:

[0368] The server converts the generated digital clone into binary format and transmits it to the terminal.

[0369] Step 9:

[0370] The device stores the received digital clone locally and activates it when the user gives the command.

[0371] Online task assistance phase

[0372] Step 10:

[0373] The user issues tasks to the digital clone through a dedicated application, such as "attend tomorrow's online meeting and take minutes."

[0374] Step 11:

[0375] The terminal analyzes the user's instructions and sends a request to the server to obtain the required information.

[0376] Step 12:

[0377] The server collects the user's schedule and meeting information and sends it to the terminal.

[0378] Step 13:

[0379] The device activates a digital clone based on the information it acquires and automatically logs in at the specified meeting time.

[0380] Step 14:

[0381] The digital clone converts speech to text in real time during meetings and creates meeting minutes, while the emotion engine monitors the user's emotional state and suggests appropriate responses.

[0382] Step 15:

[0383] After the meeting, the digital clone will compile minutes and report them to the user via their device, taking into account emotional data.

[0384] Specific examples

[0385] Example 1: Process flow for creating minutes of an online meeting

[0386] Step 1:

[0387] The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0388] Step 2:

[0389] The terminal sends the instruction to the server.

[0390] Step 3:

[0391] The server collects the details of the meeting (participant list, agenda, etc.) and sends them to the terminal.

[0392] Step 4:

[0393] A digital clone logs in at meeting time, converts speech to text in real time, and creates minutes. An emotion engine monitors the user's emotional state during the meeting.

[0394] Step 5:

[0395] The digital clone will compile minutes and report them to the user via their device after the meeting, taking into account emotional data.

[0396] Example 2: Task management process flow

[0397] Step 1:

[0398] The user says, "Organize this week's tasks and set reminders."

[0399] Step 2:

[0400] The terminal transmits the user's schedule information and emotion data to the server.

[0401] Step 3:

[0402] The server analyzes schedule and emotion data to identify important and urgent tasks.

[0403] Step 4:

[0404] The server lists the identified tasks, generates reminders based on the user's emotional state, and sends them to the device.

[0405] Step 5:

[0406] The device notifies the user of the reminder, and the emotion engine monitors the user's emotional state and adjusts the wording of the reminder accordingly.

[0407] These detailed processing steps enable the digital clone system of the present invention to efficiently perform tasks according to the user's instructions and provide appropriate support that takes into account the user's emotional state.

[0408] Example 2

[0409] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0410] While conventional digital assistant systems could assist users with online tasks using their personal data, they lacked the ability to adjust responses and behaviors based on the user's emotional state. This resulted in problems such as stress and insufficient support. Furthermore, they lacked the ability to respond in real time for specific online tasks, such as taking meeting minutes. Furthermore, prioritizing tasks and setting reminders did not take the user's emotional state into account, which meant they did not fully meet the user's needs.

[0411] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for collecting personal data and emotional data; means for encrypting the collected data and transmitting it to the server; means for the server to preprocess the received data and generate a user model using a machine learning algorithm; means for creating a digital clone based on the generated user model and emotional data and transmitting it to the terminal; and means for the digital clone to perform online tasks according to the user's instructions and adjust responses and report content using an emotion engine. This makes it possible to provide more personalized assistance by utilizing the user's personal data and emotional data, and to realize responses and actions appropriate to the user's emotional state. Furthermore, appropriate services that take the user's emotions into consideration can be provided when efficiently completing specific online tasks and prioritizing tasks.

[0412] "Personal data" refers to all information about the user, including emails, schedules, audio data, photos, internet history, and so on.

[0413] "Emotion data" refers to data that reflects the user's emotional state, and includes information such as voice and facial expressions acquired using a microphone or camera.

[0414] "Server" refers to the computer system that collects, preprocesses, analyzes, generates models, and creates digital clones of data.

[0415] "Terminal" refers to a device used by a user to operate the system, including smartphones and personal computers.

[0416] "Encryption" refers to the process of converting data into a special format for secure communication.

[0417] "Preprocessing" refers to processes such as data cleaning, filtering, and noise removal that transform raw data into a format suitable for machine learning algorithms.

[0418] A "machine learning algorithm" refers to a computational method that learns patterns from data and uses them to make predictions and classifications.

[0419] "User model" refers to a digital representation of a user's behavior and preferences that is generated based on collected personal and emotional data.

[0420] A "digital clone" refers to a software agent that is created based on a user model and is used to carry out user instructions and tasks.

[0421] "Emotion engine" refers to a function that analyzes user emotional data and utilizes that information throughout the system.

[0422] "Online tasks" refer to work or tasks carried out over the Internet, including online meetings, taking minutes, and schedule management.

[0423] The present invention relates to a system that uses a user's personal data and emotional data to create a digital clone and assist with online tasks. This system mainly consists of four components: a "server," a "terminal," a "user," and an "emotion engine." The following describes the specific means and process for implementing the present invention.

[0424] 1. Initial Setup and Data Collection

[0425] Users install the dedicated application and set up their account. Next, they select the type of data to be collected (email, schedule, voice data, photos, internet history, etc.) and set the corresponding access permissions for the application. Emotional data is collected using the microphone and camera.

[0426] For example, a user launches a dedicated smartphone app and registers the day's calendar information and schedule. At the same time, the camera captures their facial expressions and the microphone records their mood for the day.

[0427] 2. Data Preprocessing and Encryption

[0428] The device temporarily stores the collected data and performs preprocessing using the Python Pandas library to clean and denoise the data. After preprocessing, the data is encrypted using AES encryption and sent to the server using a secure protocol (e.g., HTTPS).

[0429] 3. Training and Model Generation

[0430] The server analyzes the received data and generates a user model using machine learning algorithms (e.g., scikit-learn or TensorFlow). Python libraries such as Pandas and NumPy are used for data cleaning and normalization. Emotion data is also processed through an emotion engine, which incorporates the user's emotional state into the model.

[0431] 4. Creating a Digital Clone

[0432] The server generates a digital clone based on the trained user model and emotion data. This digital clone is converted into binary format and sent to the device. The device stores the received digital clone locally and activates it when the user uses it.

[0433] 5. Assistance and execution of tasks

[0434] The user gives the digital clone instructions, such as "Take minutes of tomorrow's online meeting." The device analyzes these instructions, sends them to the server, and retrieves the necessary information. The digital clone then participates in the online meeting, converts speech to text in real time, and generates minutes. It uses an emotion engine to adjust responses and reports based on the user's emotional state.

[0435] Specific examples

[0436] Example 1: Creating minutes for online meetings

[0437] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0438] 2. The device sends this instruction to the server.

[0439] 3. The server collects the meeting details (participant list, agenda) and sends them to the terminal.

[0440] 4. The digital clone automatically logs in at the meeting time and converts the conversation into text using voice recognition. The emotion engine monitors the user's emotional state during the meeting and finalizes the minutes.

[0441] 5. The digital clone compiles the minutes and sends them to the user via the terminal.

[0442] Example 2: Task Management

[0443] 1. The user asks, "Organize this week's tasks and set reminders."

[0444] 2. The terminal sends the user's schedule information to the server.

[0445] 3. The server analyzes the schedule data and identifies important and urgent tasks, taking into account emotional data.

[0446] 4. The server lists the identified tasks and sends them to the device.

[0447] 5. The device notifies the user of the reminder. The emotion engine provides feedback on the user's emotional state and adjusts the wording of the reminder.

[0448] Prompt Sentence Examples

[0449] "Describe a system where a user provides data using a dedicated app to generate a digital clone along with emotional data."

[0450] In this way, the digital clone system of the present invention can significantly improve users' work efficiency by analyzing users' personal data and emotional data in real time and providing optimal online task support. The introduction of an emotion engine enables responses and actions that take the user's emotional state into account, providing more effective and personalized support.

[0451] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0452] Step 1:

[0453] Users install the dedicated application and set up an account. After launching the application, they select the type of data to be collected (email, schedule, voice data, photos, internet history, etc.) and set the corresponding access permissions for the application. Emotional data is also collected using the microphone and camera.

[0454] Input: User's personal and emotional data

[0455] Output: Temporarily save collected data to the device

[0456] For example, a user launches a dedicated smartphone app and allows access to calendar information and the microphone. The app then captures facial expressions with the camera and records the user's mood for the day with the microphone.

[0457] Step 2:

[0458] The terminal temporarily stores the collected data. Then, the data is cleaned to remove unnecessary noise and invalid entries. Specifically, the data is cleaned using the Python Pandas library. The data is also AES encrypted and sent to the server using a secure protocol (e.g., HTTPS).

[0459] Input: User's personal and emotional data

[0460] Output: Preprocessed encrypted data sent to the server

[0461] As a specific example, the terminal receives "schedule data" and "audio data," removes noise, applies AES encryption, and sends it to the server via HTTPS.

[0462] Step 3:

[0463] The server analyzes the received data and further cleans and normalizes it. Specifically, it processes the data using Pandas and NumPy. It then generates a user model using machine learning algorithms (e.g., scikit-learn and TensorFlow). It also analyzes the emotional data and incorporates the user's emotional state into the model via an emotion engine.

[0464] Input: Encrypted user personal and emotional data

[0465] Output: Integrated data including user model and emotional state

[0466] For example, the server normalizes large amounts of schedule data and voice data, analyzes them with machine learning algorithms, and generates a user model. Emotional data is also analyzed and reflected in the user model.

[0467] Step 4:

[0468] The server generates a digital clone based on the trained user model and emotion data. This digital clone is converted into binary format and sent to the device. The device stores the received digital clone locally and activates it when the user uses it.

[0469] Input: User model and emotion data

[0470] Output: Sending the binary digital clone to the device

[0471] As a concrete example, the server generates a digital clone based on the learned user model, encodes it as binary code, and sends it to the device, which then stores the received digital clone in local storage.

[0472] Step 5:

[0473] The user instructs the digital clone to perform an online task. For example, they give specific instructions such as, "Attend tomorrow's online meeting and take minutes." The device analyzes this instruction and sends it to the server. The server collects the necessary information and sends it back to the device. The digital clone then carries out the task and adjusts its responses and reports using an emotion engine.

[0474] Input: User instructions

[0475] Output: Task execution results and report to the user

[0476] As a concrete example, a user might say to the digital clone, "Take minutes of the meeting tomorrow at 9 o'clock." The device analyzes the instruction and sends it to the server. The server collects calendar information, and the digital clone automatically logs in to the meeting and creates minutes in real time using voice recognition. The emotion engine monitors emotions during the meeting and compiles appropriate minutes.

[0477] (Application example 2)

[0478] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0479] Conventional online task assistance systems do not take into account the user's emotional state, making it difficult to provide appropriate responses tailored to the user's actual needs and situation. Furthermore, they are unable to effectively utilize the user's preferences and past behavioral history, limiting the amount of suggestions and support they can provide. In particular, in the food delivery field, responses based on individual preferences, past behavioral history, and real-time emotional data are required to increase user satisfaction. The present invention aims to solve these problems.

[0480] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a device that collects personal data, a device that transmits the collected data to the server, a device that preprocesses the data received by the server and generates a user model using a machine learning algorithm, a device that generates a digital clone using the generated user model and transmits it to the terminal, a device that causes the digital clone to perform online tasks in accordance with the user's instructions, a device that recognizes emotional data in real time, and a device that analyzes the user's state using the recognized emotional data and suggests appropriate actions. This enables quick and accurate responses that take the user's emotional state into consideration, and in the food delivery field, it enables appropriate suggestions based on individual preferences and past behavioral history to increase user satisfaction.

[0481] "Personal data" refers to all data related to a user, such as the user's personal information, usage history, preference information, and allergy information.

[0482] A "server" is a computer system connected via a network that preprocesses data, runs machine learning algorithms, and generates user models and digital clones.

[0483] A "terminal" is a device used by a user, such as a computer or smartphone, that communicates with a server to receive a digital clone and realizes interaction with the user.

[0484] A "machine learning algorithm" is a computational method used to learn patterns and features in data and make predictions or classifications.

[0485] A "user model" is a digital profile that reflects a user's behavioral patterns and preferences, generated by machine learning algorithms.

[0486] A "digital clone" is a software agent that mimics a user's behavior and decisions, a virtual entity that performs online tasks according to the user's instructions.

[0487] "Emotion data" is data that indicates the emotional state of the user, obtained from the user's voice, facial expression, words, etc.

[0488] An "emotion engine" is a software module for recognizing and analyzing emotional data in real time.

[0489] "Online tasks" are all tasks that are performed using the Internet, and specifically include attending meetings, creating minutes, and managing schedules.

[0490] The "device for suggesting appropriate actions" is a device that recommends actions that are most suitable for the user's state based on collected emotional data and a user model.

[0491] "Food delivery" is a service that delivers food and drink to a location specified by the user.

[0492] The present invention relates to a system that uses a user's personal data and emotional data to generate a digital clone and assist with online tasks. The system collects personal data, transmits it to a server, performs data preprocessing, generates a user model using a machine learning algorithm, creates a digital clone, recognizes emotional data in real time, and suggests actions according to the user's state.

[0493] System configuration

[0494] The system includes the following components:

[0495] 1. Terminal devices for users to collect data

[0496] 2. A device where the server receives the data and generates a user model through preprocessing and machine learning algorithms.

[0497] 3. A device that uses a digital clone received by a terminal from a server to perform online tasks.

[0498] 4. A software module that enables the emotion engine to recognize and analyze emotion data in real time.

[0499] Explanation of program processing

[0500] The server receives personal and emotional data sent from the user's device. Data collection is performed using the smartphone's built-in microphone and camera, which capture voice and facial expression data. The received data is preprocessed on the server, and a user model is generated using a machine learning algorithm. This user model is a digital profile that reflects the user's past behavior and preferences. A digital clone is then generated based on this user model and sent to the user's device.

[0501] The emotion engine runs on the device and recognizes the user's real-time emotional state. The recognized emotional data is analyzed and suggestions appropriate to the user's state are made. For example, in a food delivery application, if a user is feeling stressed, the engine will suggest dishes that are suitable for relieving stress based on past preferences.

[0502] Usage example

[0503] Below are some specific examples of food delivery:

[0504] 1. The user types into the terminal, "I'm feeling stressed, so please give me some suggestions."

[0505] 2. The emotion engine detects the user's emotions in real time and recognizes "stress."

[0506] 3. The server analyzes the user's past order history and preferences based on the user model, and suggests dishes suitable for relieving stress.

[0507] 4. The device displays the suggested dishes to the user, and the user confirms the order.

[0508] 5. The server sends the order to the delivery service and manages the delivery status in real time.

[0509] Prompt Sentence Examples

[0510] An example of a prompt sentence for suggesting a dish to a user who is feeling stressed is as follows:

[0511] For example: "The user is currently feeling stressed. Their past ordering history shows that they like chocolate cake and comfort food to relieve stress. Please recommend appropriate dishes based on their current emotional state."

[0512] As described above, the system of the present invention can provide a fast and accurate response that takes into account the user's emotional state, which can increase user satisfaction, especially in the food delivery field. It can also provide high-level support using emotional data for other online tasks.

[0513] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0514] Step 1:

[0515] User collects data

[0516] Users use their smartphone's camera and microphone to collect personal and emotional data. Input data includes voice and facial expression data. This data is temporarily stored on the device. The collected voice and facial expression data is provided as output.

[0517] Step 2:

[0518] The device sends the data to the server

[0519] The device encrypts the collected personal data and emotion data and transmits them to the server via a secure protocol. The input data includes the collected voice and facial expression data. The output data includes the personal data and emotion data received by the server.

[0520] Step 3:

[0521] The server preprocesses the data

[0522] The server preprocesses the received data, removing noise and normalizing the data. The input data includes the server's personal data and emotion data sent from the device. After preprocessing, the cleaned data is stored on the server as the output.

[0523] Step 4:

[0524] The server generates the user model

[0525] The server uses the preprocessed data to generate a user model based on machine learning algorithms. The input data includes the preprocessed personal data and emotional data. The output is a user model, which is a digital profile that reflects the user's behavioral patterns and preferences.

[0526] Step 5:

[0527] The server creates a digital clone and sends it to the device.

[0528] The server creates a digital clone based on the generated user model and sends it to the terminal. The input data includes the user model. The output data includes the digital clone received by the terminal.

[0529] Step 6:

[0530] Emotion engine recognizes emotional data in real time

[0531] The emotion engine of the device recognizes the user's emotion data in real time. The input data includes real-time emotion data obtained from the user's voice and facial expressions. The output is the detected user's emotional state provided to the device.

[0532] Step 7:

[0533] The emotion engine analyzes the user's emotional state and makes appropriate suggestions to them.

[0534] The emotion engine on the device analyzes the recognized emotion data and the user model and makes suggestions appropriate to the user's state. For example, if a user is feeling stressed, it will suggest dishes that are good for relieving stress. The input data includes the detected emotional state and the user model. The output is a display of the suggestions made to the user.

[0535] Step 8:

[0536] The user confirms the order

[0537] The user confirms the order based on the suggestions displayed on the terminal. The input data includes the suggestions displayed on the terminal. The output is the confirmed order sent to the server.

[0538] Step 9:

[0539] The server processes the order and sends it to the delivery service

[0540] The server processes the user's confirmed order and sends it to the delivery service. The input data includes the user's confirmed order. The output data includes the order details received by the delivery service.

[0541] Step 10:

[0542] The server manages the delivery status

[0543] The server tracks the delivery status from the delivery service in real time and provides the information to the user. The input data includes delivery progress information from the delivery service. The output data includes delivery progress information provided to the user.

[0544] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0545] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0546] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0547] [Second embodiment]

[0548] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0549] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0550] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0551] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0552] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0553] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0554] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0555] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0556] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0557] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0558] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0559] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0560] The present invention relates to a system for creating a digital clone using personal data of a user and assisting with online tasks. Specific embodiments of the system are described below.

[0561] Digital Clone System Overview

[0562] The system of the present invention is composed of three main components: a "server," a "terminal," and a "user." Users interact with the system using dedicated applications. Data collection, processing, cloning, and task assistance are performed, and each step to achieve this is described in detail below.

[0563] User Data Collection

[0564] Users provide their own data, including email history, schedules, voice data, photos, internet history, etc. Users select this data through a dedicated application and upload it to the system.

[0565] The device temporarily stores this data and transmits it to a server in encrypted form using a secure protocol.

[0566] Data Preprocessing and Training

[0567] The server preprocesses the received data, which includes cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences and generate a user model. This model serves as the basis for creating a digital clone of the user.

[0568] Creating a digital clone

[0569] The server generates a digital clone based on the learned user model. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions.

[0570] The device stores the received digital clone locally and activates it when the user wants to use it.

[0571] Online task assistance

[0572] The user instructs the digital clone to perform online tasks, such as "attend tomorrow's online meeting and take minutes."

[0573] The device analyzes this instruction and requests the necessary information from the server, which then collects meeting information from the user's schedule management system and sends the details to the device.

[0574] The digital clone automatically logs in at the designated meeting time, performs real-time voice recognition, and converts the conversation into text. When the meeting ends, a summary of the meeting minutes is sent to the user via the device.

[0575] Specific examples

[0576] Example 1: Creating minutes for online meetings

[0577] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0578] 2. The device sends this instruction to the server.

[0579] 3. The server collects the meeting details (participant list, agenda, etc.) and sends them to the terminal.

[0580] 4. A digital clone logs in at the meeting time and recognizes and transcribes the conversation in real time.

[0581] 5. The digital clone compiles minutes and sends them to the user via the terminal after the meeting ends.

[0582] Example 2: Task Management

[0583] 1. The user asks, "Organize this week's tasks and set reminders."

[0584] 2. The device sends the user's schedule to the server.

[0585] 3. The server analyzes the user's schedule and identifies high-priority tasks.

[0586] 4. The server lists the identified tasks, sets reminders and sends them to the device.

[0587] 5. The device notifies the user of the reminder.

[0588] How to use

[0589] To use the system of the present invention, a user installs a dedicated application and registers as a user. Next, the user selects and provides the data they want to collect, and begins creating a digital clone. Once the digital clone is created, the user can instruct the digital clone to perform various online tasks through the dedicated application, improving work efficiency.

[0590] In this way, by implementing the digital clone system of the present invention, users can efficiently manage and carry out their own work, thereby reducing their workload.

[0591] The processing flow will be explained below.

[0592] Digital clone system processing flow

[0593] Data Collection Phase

[0594] Step 1:

[0595] The user launches the dedicated application and selects the data they want to collect (emails, schedules, audio data, photos, internet history, etc.).

[0596] Step 2:

[0597] The device temporarily stores selected data, for example, email history in a temporary folder in local storage.

[0598] Step 3:

[0599] The device encrypts the stored data using a secure communication protocol (e.g., SSL / TLS) and sends it to the server.

[0600] Data preprocessing and training phase

[0601] Step 4:

[0602] The server preprocesses the data it receives. Specifically, it cleans and filters the data to remove unnecessary noise. For example, it removes spam and advertising emails from email data.

[0603] Step 5:

[0604] The server feeds the preprocessed data into machine learning algorithms to generate a user model, a process that learns and models the user's behavioral patterns and preferences.

[0605] Digital Clone Generation Phase

[0606] Step 6:

[0607] The server generates a digital clone based on the learned user model. This digital clone is an agent that simulates the user's actions and decisions.

[0608] Step 7:

[0609] The server converts the generated digital clone into binary format and transmits it to the terminal.

[0610] Step 8:

[0611] The device stores the received digital clone locally and launches it when the user needs it.

[0612] Online task assistance phase

[0613] Step 9:

[0614] The user gives the digital clone tasks through a dedicated application, such as "attend tomorrow's online meeting and take minutes."

[0615] Step 10:

[0616] The terminal analyzes the user's instructions and sends a request to the server to obtain the required information.

[0617] Step 11:

[0618] The server collects the user's schedule and meeting information and sends it to the device, for example extracting meeting details from the user's calendar.

[0619] Step 12:

[0620] Based on the information acquired by the device, a digital clone is activated and allowed to participate in the meeting. The digital clone converts speech into text in real time and creates minutes.

[0621] Step 13:

[0622] After the meeting, the digital clone compiles minutes and sends them to the user via the device.

[0623] Step 14:

[0624] The user reviews the minutes they receive and gives the digital clone further instructions, such as "Share the minutes with the team."

[0625] Specific examples

[0626] Example 1: Process flow for creating minutes of an online meeting

[0627] Step 1:

[0628] The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0629] Step 2:

[0630] The terminal analyzes the instructions and sends them to the server.

[0631] Step 3:

[0632] The server collects the details of the meeting (participant list, agenda, etc.) and sends them to the terminal.

[0633] Step 4:

[0634] The digital clone automatically logs in at meeting time and recognizes voice and transcribes the conversation in real time.

[0635] Step 5:

[0636] The digital clone compiles minutes and sends them to the user via the terminal after the meeting ends.

[0637] Example 2: Task management process flow

[0638] Step 1:

[0639] The user says, "Organize this week's tasks and set reminders."

[0640] Step 2:

[0641] The terminal transmits the user's schedule information to the server.

[0642] Step 3:

[0643] The server analyzes the schedule and identifies important and urgent tasks.

[0644] Step 4:

[0645] The server lists the identified tasks and sends them to the terminal.

[0646] Step 5:

[0647] The device notifies the user of the reminder.

[0648] These detailed processing steps allow the digital clone system of the present invention to efficiently perform tasks according to the user's instructions, thereby reducing the burden on the user.

[0649] Example 1

[0650] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0651] In today's world, with the increase in online tasks, it is becoming increasingly difficult for individual users to efficiently manage their time and complete tasks. Furthermore, taking minutes of online meetings and managing tasks is time-consuming and accuracy is an issue. This has led users to seek ways to improve the efficiency of their work.

[0652] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0653] In this invention, the server includes means for collecting personal data, means for transmitting the collected data to the server using an encryption protocol, means for the server to preprocess the received data and generate a user model using a machine learning algorithm, means for the server to create a digital clone using the generated user model and transmit it to the terminal, means for the digital clone to analyze the user's instructions using natural language processing and execute online tasks, and means for the digital clone to notify the user of the results after the task is executed. This enables users to efficiently manage and execute online tasks, significantly improving work efficiency.

[0654] "Personal data" refers to any form of information related to a user (email history, schedule, audio data, photos, internet history, etc.).

[0655] "Cryptographic protocol" refers to the technology (e.g., TLS) used to ensure the secure communication of data.

[0656] A "server" refers to a computer system that performs core computations, such as data preprocessing, generating machine learning models, and creating digital clones.

[0657] A "machine learning algorithm" refers to a computational method that learns patterns from data and generates and updates a user model.

[0658] A "user model" refers to a data structure that represents a user's behavioral patterns and preferences, generated by a machine learning algorithm based on collected personal data.

[0659] A "digital clone" refers to a software agent that is generated based on a user model and mimics the user's behavior and decisions.

[0660] "Terminal" refers to the device (e.g., smartphone, tablet, PC) through which a user interacts with their digital clone.

[0661] "Natural language processing" refers to the technology that a digital clone uses to understand and interpret user instructions.

[0662] "Online tasks" refer to Internet tasks that a user instructs a digital clone to perform (e.g., participating in an online meeting, task management).

[0663] "Speech recognition technology" refers to technology that converts speech into text in real time.

[0664] "Minutes" refers to a document that records the contents of the conversation that took place during an online meeting as text.

[0665] "Reminder" refers to a function that notifies the user according to a schedule.

[0666] "Information collection system" refers to external APIs and tools for collecting user schedules and other necessary information.

[0667] The present invention relates to a system for creating a digital clone using personal data of a user and assisting with online tasks. Specific embodiments of the system are described below.

[0668] User Data Collection

[0669] Users use a dedicated application to provide their own data, including email history, schedules, voice data, photos, internet history, etc. The user selects this data and uploads it to the system.

[0670] The device temporarily stores this data and transmits it to the server using an encryption protocol such as TLS (Transport Layer Security).

[0671] Data Preprocessing and Training

[0672] The server preprocesses the received data, including cleaning the data (e.g., imputing missing values ​​and removing outliers), filtering, and normalization. This preprocessing is performed using Python's Pandas and Numpy libraries.

[0673] The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences, generating a user model using libraries like TensorFlow and PyTorch. This user model serves as the basis for creating a digital clone of the user.

[0674] Creating a digital clone

[0675] The server generates a digital clone based on the learned user model. This digital clone is a software agent generated using Python scripts that mimics the user's behavior and decisions. The generated digital clone is converted into a binary format and sent to the device.

[0676] The terminal stores the received digital clone in local storage, using SQLite or JSON format. When the user wants to use it, the terminal launches the digital clone, possibly using a Python script or a Docker container.

[0677] Online task assistance

[0678] The user instructs the digital clone to perform online tasks. These instructions are entered as prompt sentences through the interface of a dedicated application. For example, the user might instruct the clone to "attend tomorrow's online meeting and take minutes."

[0679] The device parses this instruction using natural language processing (NLP) techniques (e.g., spaCy or NLTK libraries). The device then requests the necessary information from the server. For example, it could retrieve meeting information from the Google Calendar API.

[0680] The server collects meeting information from the user's scheduler and sends the details to the device. The digital clone then automatically logs in at the designated meeting time and performs real-time speech recognition and converts the conversation into text, using, for example, the Google Cloud Speech-to-Text API. Once the meeting is over, a summary of the meeting minutes is reported to the user via the device.

[0681] Specific examples

[0682] Example 1: Creating minutes for online meetings

[0683] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0684] 2. The device analyzes this instruction and requests the necessary information from the server.

[0685] 3. The server uses the Google Calendar API to collect the meeting details and send them to the device.

[0686] 4. At the meeting time, the digital clone logs in using Selenium and converts the audio to text using the Google Cloud Speech-to-Text API.

[0687] 5. The digital clone compiles the minutes in Markdown format and notifies the user through a dedicated application.

[0688] Example 2: Task Management

[0689] 1. The user asks, "Organize this week's tasks and set reminders."

[0690] 2. The terminal sends the user's schedule information to the server.

[0691] 3. The server uses Natural Language Processing (NLP) technology to analyze the schedule and identify high-priority tasks.

[0692] 4. The server lists the identified tasks, sets reminders using the Google Calendar API, and sends them to the device.

[0693] 5. The device notifies the user of the reminder.

[0694] Prompt Sentence Examples

[0695] "Please take the minutes of the meeting tomorrow at 9 o'clock."

[0696] "Organize my tasks for this week and set reminders."

[0697] By following this specific process flow, users can efficiently manage and complete online tasks.

[0698] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0699] Step 1:

[0700] The user launches a dedicated application and provides their personal data (e.g., email history, schedule, audio data, photos, internet history).

[0701] Input: Personal Data

[0702] Output: Temporarily saved dataset

[0703] Specific operations: Select the type of data to provide through the dedicated application interface and press the button to upload the data.

[0704] Step 2:

[0705] The device temporarily stores the data uploaded by the user and sends it to the server using an encryption protocol such as TLS (Transport Layer Security).

[0706] Input: Temporarily saved dataset

[0707] Output: Data in encrypted form

[0708] Specific operation: The data is temporarily stored in a local database (e.g., SQLite) on the device, and then encrypted using the OpenSSL library. After encryption, the data is sent to the server.

[0709] Step 3:

[0710] The server stores the received data in a cloud database (e.g., Amazon RDS) and performs preprocessing.

[0711] Input: Encrypted data

[0712] Output: Preprocessed dataset

[0713] Specific operations: Decrypt the data and use the Pandas library to clean (fill in missing values, remove outliers), filter, and normalize the data.

[0714] Step 4:

[0715] The server inputs the preprocessed data into a machine learning algorithm (e.g., TensorFlow, PyTorch) to generate a user model.

[0716] Input: Preprocessed dataset

[0717] Output: User model

[0718] Specific operation: Data is fed into a machine learning algorithm to learn behavioral patterns and preferences. As a result, a user model is generated.

[0719] Step 5:

[0720] The server generates a digital clone based on the user model, converts it into binary format, and transmits it to the terminal.

[0721] Input: User model

[0722] Output: Digital clone in binary format

[0723] Specific operation: The user model is embodied as a digital clone using a Python script, then converted into binary format. The converted digital clone is sent to the terminal.

[0724] Step 6:

[0725] The terminal stores the received digital clone in local storage (e.g., SQLite, JSON format) and activates the digital clone as needed.

[0726] Input: Digital clone in binary format

[0727] Output: Activated digital clone

[0728] Specific operation: A digital clone is stored in local storage and launched using a Python script or Docker container when the user uses it.

[0729] Step 7:

[0730] The user instructs the digital clone on tasks through a dedicated application.

[0731] Input: Prompt sentence (e.g., "Please attend tomorrow's online meeting and take minutes.")

[0732] Output: Parsed task instructions

[0733] Specific operations: Enter the prompt text through the dedicated application interface and press the send button.

[0734] Step 8:

[0735] The device analyzes the user's instructions using natural language processing (NLP) technology (e.g., spaCy, NLTK) and requests the necessary information from the server.

[0736] Input: prompt statement

[0737] Output: Task execution information request

[0738] Specific operations: Analyzes the text of the user's instructions using NLP technology, interprets the task content, and sends a request to the server for the appropriate information.

[0739] Step 9:

[0740] The server collects the necessary information from the user's schedule management system (e.g., Google Calendar API) and sends it to the device.

[0741] Input: Task execution information request

[0742] Output: Collected schedule information

[0743] Specific operation: Uses API to obtain user schedule information and sends it to the device.

[0744] Step 10:

[0745] The digital clone performs online tasks (e.g., participating in an online meeting) and notifies the user of the results.

[0746] Input: Collected schedule information

[0747] Output: Task results (e.g. meeting minutes)

[0748] Specific operations: Log in at the specified meeting time (e.g., using Selenium), convert the audio into text using the Google Cloud Speech-to-Text API, summarize the task results, and notify the user.

[0749] Prompt Sentence Examples

[0750] "Please take the minutes of the meeting tomorrow at 9 o'clock."

[0751] "Organize my tasks for this week and set reminders."

[0752] The above is the specific processing flow of this system.

[0753] (Application example 1)

[0754] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0755] While existing technologies exist that generate digital clones based on a user's personal data and assist with online tasks, they lack the means to analyze a customer's purchase history and preferences in physical stores and recommend appropriate products. As a result, improvements to customer service and the shopping experience in physical stores have not been fully realized. The present invention aims to solve this problem and improve the customer experience in physical stores.

[0756] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0757] In this invention, the server includes a means for acquiring personal data, a means for transmitting the acquired data to the server, a means for the server to preprocess the received data and generate a user model using a machine learning algorithm, a means for creating a digital clone using the generated user model and transmitting it to the terminal, a means for the digital clone to perform online tasks according to the user's instructions, and a means for the digital clone to analyze the customer's purchase history and preferences using smart glasses in a physical store and recommend appropriate products. This enables the digital clone to immediately respond to customer needs even in a physical store and recommend optimal products.

[0758] "Personal data" refers to data that includes a user's behavioral history, purchasing history, preferences, and other personal information.

[0759] A "server" is a computer system that receives user data, pre-processes it, generates models using machine learning, and creates and transmits digital clones.

[0760] A "terminal" is a device that receives a digital clone sent from a server and allows a user to interact with the digital clone.

[0761] A "machine learning algorithm" is a technology that analyzes user data and learns user behavior patterns and preferences.

[0762] A "user model" is a data model that reflects a user's behavioral patterns and preferences, generated using a machine learning algorithm.

[0763] A "digital clone" is a software agent that mimics a user's behavior and decisions based on a generated user model.

[0764] An "online task" is a task performed over the Internet and executed by the digital clone based on the user's instructions.

[0765] A "physical store" is a physically existing sales store that is a place that users (customers) visit in person.

[0766] "Smart glasses" are wearable devices that have augmented reality and information display functions and are used by store clerks in physical stores when providing customer service.

[0767] "Purchase history" is a record of products that a customer has purchased in the past.

[0768] "Preferences" refer to a user's tendency to prefer a particular type or style of product.

[0769] In the context of "Levidence," a "designated document" is a document submitted in the course of a trial or investigation that is referenced based on specific requirements.

[0770] The present invention is a system for collecting personal data of users and generating digital clones, and it is particularly intended to provide personalized product recommendations to customers using smart glasses in brick-and-mortar stores. Specific embodiments of the system are described below.

[0771] System Overview

[0772] The system of the present invention consists of a server, a terminal, and a user. The user interacts with the system using smart glasses, and data collection, processing, digital cloning, and product recommendations are performed. This procedure is described in detail below.

[0773] Retrieving User Data

[0774] Users provide their own data, including purchase history, preferences, and other behavioral patterns. They select this data through a dedicated application and upload it to the system. The device temporarily stores this data and transmits it to the server in encrypted form using a secure protocol.

[0775] Data preprocessing and user model generation

[0776] The server preprocesses the received data, which includes cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms (e.g., the scikit-learn library) to learn the user's behavioral patterns and preferences and generate a user model.

[0777] Creating and transmitting digital clones

[0778] The server generates a digital clone based on the learned user model. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions. The device stores the received digital clone locally and activates it when the user uses it.

[0779] Product recommendations in physical stores

[0780] A user wears smart glasses and performs the task of recommending products to a customer. For example, a customer may request, "Please show me the latest new products." The device sends this instruction to the server, which analyzes the customer's purchase history and preferences. Based on the analysis results, the device sends a recommendation list of appropriate new products. The user can then suggest the best products to the customer based on the information displayed on the smart glasses' display.

[0781] Specific examples

[0782] Usage Scenarios

[0783] 1. Customer A asks the store clerk to show him the latest new products.

[0784] 2. The device sends this instruction to the server.

[0785] 3. The server analyzes customer A's purchase history and preferences, generates a list of recommended products, and sends it to the terminal.

[0786] 4. The user (store clerk) introduces the recommended product to customer A through the smart glasses.

[0787] Prompt Sentence Examples

[0788] "Customer ID: customer123. Looking at their recent purchase history, they have purchased a bag, shoes, and a jacket. New items have arrived. What products would you recommend for this customer?"

[0789] Hardware and software used

[0790] Hardware: Servers (e.g., Amazon Web Services, Google Cloud Platform)

[0791] Software: Python libraries (e.g., requests, scikit-learn)

[0792] This will enable the digital clone to instantly respond to customer needs in physical stores and recommend the most suitable products.

[0793] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0794] Step 1:

[0795] User provides data

[0796] Users select data such as their purchase history, preferences, and behavioral patterns through a dedicated application and upload that data to the device. INPUT is the user's personal information and history data, and OUTPUT is encrypted data stored on the device.

[0797] Step 2:

[0798] The device sends data to the server

[0799] The device temporarily stores the acquired data and sends it to the server in encrypted form using a secure protocol (e.g., HTTPS). INPUT is the encrypted data stored on the device, and OUTPUT is the encrypted data sent to the server.

[0800] Step 3:

[0801] The server preprocesses the data

[0802] The server performs pre-processing on the received data, such as cleaning, filtering, and normalization. Unwanted noise is removed and the data is in a format that can be analyzed. INPUT is the encrypted data sent to the server, and OUTPUT is the pre-processed data.

[0803] Step 4:

[0804] The server generates a user model using a machine learning algorithm

[0805] The server uses the preprocessed data to apply machine learning algorithms (e.g., scikit-learn) to learn the user's behavioral patterns and preferences, which results in a user model. The input is the preprocessed data, and the output is the user model.

[0806] Step 5:

[0807] The server uses the user model to create a digital clone

[0808] The server creates a digital clone based on the generated user model. The created digital clone is converted into binary format and sent to the terminal. The input is the user model, and the output is the binary format digital clone.

[0809] Step 6:

[0810] The device stores the digital clone locally and activates it when the user wants to use it.

[0811] The terminal stores the received digital clone locally and activates it as needed. This allows the user to receive services from the digital clone. INPUT is the digital clone sent from the server, and OUTPUT is the digital clone stored in the terminal.

[0812] Step 7:

[0813] Users wear smart glasses to serve customers

[0814] The user (store clerk) wears smart glasses and receives requests from customers (e.g., "Please show me the latest new products"). INPUT is the customer's request, and OUTPUT is the instruction sent to the terminal.

[0815] Step 8:

[0816] The terminal sends the customer's request to the server

[0817] The terminal analyzes the customer's request and sends the information to the server. INPUT is the customer's request, and OUTPUT is the request to the server.

[0818] Step 9:

[0819] The server analyzes the customer's purchase history and preferences to recommend appropriate products.

[0820] The server analyzes the customer's purchase history and preferences based on the received request. This generates an appropriate product recommendation list and sends it to the terminal. INPUT is the request to the server and customer data, and OUTPUT is the list of recommended products.

[0821] Step 10:

[0822] Users can suggest products to customers through smart glasses

[0823] The user (store clerk) recommends the best products to the customer based on the product recommendation list displayed on the smart glasses display. The input is the list of recommended products, and the output is the product proposal to the customer.

[0824] This allows the digital clone to instantly respond to customer needs in a physical store and recommend the most suitable products.

[0825] "Customer ID: customer123. Looking at their recent purchase history, they have purchased a bag, shoes, and a jacket. New items have arrived. What products would you recommend for this customer?"

[0826] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0827] The present invention relates to a system for creating a digital clone using personal data and emotional data of a user and assisting the user in online tasks. Specific embodiments of the system are described below.

[0828] Digital Clone System Overview

[0829] The system of the present invention consists of four main components: a "server," a "terminal," a "user," and an "emotion engine." The user interacts with the system using a dedicated application, which collects data, processes it, creates clones, and assists with tasks. The emotion engine also recognizes the user's emotions in real time and utilizes this information throughout the system.

[0830] User and sentiment data collection

[0831] The user selects the data they want to collect (email, schedule, voice data, photos, internet history, etc.) through a dedicated application, and simultaneously collects emotional data (voice and facial expression data). Emotional data is acquired through sensors such as a microphone and camera.

[0832] The device temporarily stores this data and transmits it to a server in encrypted form using a secure protocol.

[0833] Data Preprocessing and Training

[0834] The server preprocesses the received data. This involves cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences and generate a user model. This model serves as the basis for creating a digital clone of the user.

[0835] The server similarly processes the emotion data obtained through the emotion engine and incorporates the user's emotional state into the model.

[0836] Creating a digital clone

[0837] The server generates a digital clone based on the trained user model and emotional data. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions, and responds taking into account the user's emotional state.

[0838] The device stores the received digital clone locally and activates it when the user wants to use it.

[0839] Online task assistance

[0840] The user instructs the digital clone to perform online tasks, such as "attend tomorrow's online meeting and take minutes."

[0841] The device analyzes this instruction and requests the necessary information from the server, which then collects meeting information from the user's schedule management system and sends the details to the device.

[0842] The digital clone automatically logs in at the designated meeting time, performs real-time speech recognition, and converts the conversation into text. When the meeting ends, a summary of the meeting minutes is sent to the user via their device. In addition, an emotion engine monitors the user's emotional state during the meeting and creates appropriate responses and meeting minutes based on that information.

[0843] Specific examples

[0844] Example 1: Creating minutes for online meetings

[0845] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0846] 2. The device sends this instruction to the server.

[0847] 3. The server collects the meeting details (participant list, agenda, etc.) and sends them to the terminal.

[0848] 4. The digital clone automatically logs in at the meeting time, recognizes the conversation in real time, and transcribes it into text. The emotion engine simultaneously monitors the person's emotional state.

[0849] 5. The digital clone compiles minutes of the meeting, taking into account emotional data, and sends them to the user via their device after the meeting ends.

[0850] Example 2: Task Management

[0851] 1. The user asks, "Organize this week's tasks and set reminders."

[0852] 2. The terminal sends the user's schedule information to the server.

[0853] 3. The server analyzes the schedule and identifies important and urgent tasks, taking into account emotional data.

[0854] 4. The server lists the identified tasks and sends them to the device.

[0855] 5. The device notifies the user of the reminder. The emotion engine provides feedback on the user's emotional state and adjusts the wording of the reminder.

[0856] How to use

[0857] To use the system of the present invention, a user installs a dedicated application and registers as a user. Next, the user selects and provides the data they want to collect, and begins creating a digital clone. Once the digital clone is created, the user can instruct the digital clone to perform various online tasks through the dedicated application, improving work efficiency.

[0858] Furthermore, the introduction of an emotion engine enables the digital clone to respond and act in a way that takes into account the user's emotional state, providing more appropriate and effective support.

[0859] In this way, by implementing the digital clone system of the present invention, users can efficiently manage and carry out their own work, reducing their workload. The introduction of an emotion engine will provide even more advanced support.

[0860] The processing flow will be explained below.

[0861] Digital clone system processing flow

[0862] Data Collection Phase

[0863] Step 1:

[0864] The user launches the dedicated application and selects the data they want to collect (emails, schedules, audio data, photos, internet history, etc.).

[0865] Step 2:

[0866] The device temporarily stores the selected data, while simultaneously collecting the user's voice and facial expression data using the microphone and camera.

[0867] Step 3:

[0868] The device analyzes the emotional state using voice and facial expression data and generates emotional data.

[0869] Step 4:

[0870] The personal data and emotional data collected by the device are sent to a server using a secure protocol (e.g., encryption using SSL / TLS).

[0871] Data preprocessing and training phase

[0872] Step 5:

[0873] The server preprocesses the received personal and emotional data, specifically cleaning, filtering, and denoising the data.

[0874] Step 6:

[0875] The server then feeds the pre-processed data into machine learning algorithms to generate a user model that reflects the user's behavioral patterns and emotional state.

[0876] Digital Clone Generation Phase

[0877] Step 7:

[0878] The server generates a digital clone based on the learned user model and emotional data. The digital clone simulates the user's behavior and decisions and responds according to the user's emotions.

[0879] Step 8:

[0880] The server converts the generated digital clone into binary format and transmits it to the terminal.

[0881] Step 9:

[0882] The device stores the received digital clone locally and activates it when the user gives the command.

[0883] Online task assistance phase

[0884] Step 10:

[0885] The user issues tasks to the digital clone through a dedicated application, such as "attend tomorrow's online meeting and take minutes."

[0886] Step 11:

[0887] The terminal analyzes the user's instructions and sends a request to the server to obtain the required information.

[0888] Step 12:

[0889] The server collects the user's schedule and meeting information and sends it to the terminal.

[0890] Step 13:

[0891] The device activates a digital clone based on the information it acquires and automatically logs in at the specified meeting time.

[0892] Step 14:

[0893] The digital clone converts speech to text in real time during meetings and creates meeting minutes, while the emotion engine monitors the user's emotional state and suggests appropriate responses.

[0894] Step 15:

[0895] After the meeting, the digital clone will compile minutes and report them to the user via their device, taking into account emotional data.

[0896] Specific examples

[0897] Example 1: Process flow for creating minutes of an online meeting

[0898] Step 1:

[0899] The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0900] Step 2:

[0901] The terminal sends the instruction to the server.

[0902] Step 3:

[0903] The server collects the details of the meeting (participant list, agenda, etc.) and sends them to the terminal.

[0904] Step 4:

[0905] A digital clone logs in at meeting time, converts speech to text in real time, and creates minutes. An emotion engine monitors the user's emotional state during the meeting.

[0906] Step 5:

[0907] The digital clone will compile minutes and report them to the user via their device after the meeting, taking into account emotional data.

[0908] Example 2: Task management process flow

[0909] Step 1:

[0910] The user says, "Organize this week's tasks and set reminders."

[0911] Step 2:

[0912] The terminal transmits the user's schedule information and emotion data to the server.

[0913] Step 3:

[0914] The server analyzes schedule and emotion data to identify important and urgent tasks.

[0915] Step 4:

[0916] The server lists the identified tasks, generates reminders based on the user's emotional state, and sends them to the device.

[0917] Step 5:

[0918] The device notifies the user of the reminder, and the emotion engine monitors the user's emotional state and adjusts the wording of the reminder accordingly.

[0919] These detailed processing steps enable the digital clone system of the present invention to efficiently perform tasks according to the user's instructions and provide appropriate support that takes into account the user's emotional state.

[0920] Example 2

[0921] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0922] While conventional digital assistant systems could assist users with online tasks using their personal data, they lacked the ability to adjust responses and behaviors based on the user's emotional state. This resulted in problems such as stress and insufficient support. Furthermore, they lacked the ability to respond in real time for specific online tasks, such as taking meeting minutes. Furthermore, prioritizing tasks and setting reminders did not take the user's emotional state into account, which meant they did not fully meet the user's needs.

[0923] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for collecting personal data and emotional data; means for encrypting the collected data and transmitting it to the server; means for the server to preprocess the received data and generate a user model using a machine learning algorithm; means for creating a digital clone based on the generated user model and emotional data and transmitting it to the terminal; and means for the digital clone to perform online tasks according to the user's instructions and adjust responses and report content using an emotion engine. This makes it possible to provide more personalized assistance by utilizing the user's personal data and emotional data, and to realize responses and actions appropriate to the user's emotional state. Furthermore, appropriate services that take the user's emotions into consideration can be provided when efficiently completing specific online tasks and prioritizing tasks.

[0924] "Personal data" refers to all information about the user, including emails, schedules, audio data, photos, internet history, and so on.

[0925] "Emotion data" refers to data that reflects the user's emotional state, and includes information such as voice and facial expressions acquired using a microphone or camera.

[0926] "Server" refers to the computer system that collects, preprocesses, analyzes, generates models, and creates digital clones of data.

[0927] "Terminal" refers to a device used by a user to operate the system, including smartphones and personal computers.

[0928] "Encryption" refers to the process of converting data into a special format for secure communication.

[0929] "Preprocessing" refers to processes such as data cleaning, filtering, and noise removal that transform raw data into a format suitable for machine learning algorithms.

[0930] A "machine learning algorithm" refers to a computational method that learns patterns from data and uses them to make predictions and classifications.

[0931] "User model" refers to a digital representation of a user's behavior and preferences that is generated based on collected personal and emotional data.

[0932] A "digital clone" refers to a software agent that is created based on a user model and is used to carry out user instructions and tasks.

[0933] "Emotion engine" refers to a function that analyzes user emotional data and utilizes that information throughout the system.

[0934] "Online tasks" refer to work or tasks carried out over the Internet, including online meetings, taking minutes, and schedule management.

[0935] The present invention relates to a system that uses a user's personal data and emotional data to create a digital clone and assist with online tasks. This system mainly consists of four components: a "server," a "terminal," a "user," and an "emotion engine." The following describes the specific means and process for implementing the present invention.

[0936] 1. Initial Setup and Data Collection

[0937] Users install the dedicated application and set up their account. Next, they select the type of data to be collected (email, schedule, voice data, photos, internet history, etc.) and set the corresponding access permissions for the application. Emotional data is collected using the microphone and camera.

[0938] For example, a user launches a dedicated smartphone app and registers the day's calendar information and schedule. At the same time, the camera captures their facial expressions and the microphone records their mood for the day.

[0939] 2. Data Preprocessing and Encryption

[0940] The device temporarily stores the collected data and performs preprocessing using the Python Pandas library to clean and denoise the data. After preprocessing, the data is encrypted using AES encryption and sent to the server using a secure protocol (e.g., HTTPS).

[0941] 3. Training and Model Generation

[0942] The server analyzes the received data and generates a user model using machine learning algorithms (e.g., scikit-learn or TensorFlow). Python libraries such as Pandas and NumPy are used for data cleaning and normalization. Emotion data is also processed through an emotion engine, which incorporates the user's emotional state into the model.

[0943] 4. Creating a Digital Clone

[0944] The server generates a digital clone based on the trained user model and emotion data. This digital clone is converted into binary format and sent to the device. The device stores the received digital clone locally and activates it when the user uses it.

[0945] 5. Assistance and execution of tasks

[0946] The user gives the digital clone instructions, such as "Take minutes of tomorrow's online meeting." The device analyzes these instructions, sends them to the server, and retrieves the necessary information. The digital clone then participates in the online meeting, converts speech to text in real time, and generates minutes. It uses an emotion engine to adjust responses and reports based on the user's emotional state.

[0947] Specific examples

[0948] Example 1: Creating minutes for online meetings

[0949] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[0950] 2. The device sends this instruction to the server.

[0951] 3. The server collects the meeting details (participant list, agenda) and sends them to the terminal.

[0952] 4. The digital clone automatically logs in at the meeting time and converts the conversation into text using voice recognition. The emotion engine monitors the user's emotional state during the meeting and finalizes the minutes.

[0953] 5. The digital clone compiles the minutes and sends them to the user via the terminal.

[0954] Example 2: Task Management

[0955] 1. The user asks, "Organize this week's tasks and set reminders."

[0956] 2. The terminal sends the user's schedule information to the server.

[0957] 3. The server analyzes the schedule data and identifies important and urgent tasks, taking into account emotional data.

[0958] 4. The server lists the identified tasks and sends them to the device.

[0959] 5. The device notifies the user of the reminder. The emotion engine provides feedback on the user's emotional state and adjusts the wording of the reminder.

[0960] Prompt Sentence Examples

[0961] "Describe a system where a user provides data using a dedicated app to generate a digital clone along with emotional data."

[0962] In this way, the digital clone system of the present invention can significantly improve users' work efficiency by analyzing users' personal data and emotional data in real time and providing optimal online task support. The introduction of an emotion engine enables responses and actions that take the user's emotional state into account, providing more effective and personalized support.

[0963] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0964] Step 1:

[0965] Users install the dedicated application and set up an account. After launching the application, they select the type of data to be collected (email, schedule, voice data, photos, internet history, etc.) and set the corresponding access permissions for the application. Emotional data is also collected using the microphone and camera.

[0966] Input: User's personal and emotional data

[0967] Output: Temporarily save collected data to the device

[0968] For example, a user launches a dedicated smartphone app and allows access to calendar information and the microphone. The app then captures facial expressions with the camera and records the user's mood for the day with the microphone.

[0969] Step 2:

[0970] The terminal temporarily stores the collected data. Then, the data is cleaned to remove unnecessary noise and invalid entries. Specifically, the data is cleaned using the Python Pandas library. The data is also AES encrypted and sent to the server using a secure protocol (e.g., HTTPS).

[0971] Input: User's personal and emotional data

[0972] Output: Preprocessed encrypted data sent to the server

[0973] As a specific example, the terminal receives "schedule data" and "audio data," removes noise, applies AES encryption, and sends it to the server via HTTPS.

[0974] Step 3:

[0975] The server analyzes the received data and further cleans and normalizes it. Specifically, it processes the data using Pandas and NumPy. It then generates a user model using machine learning algorithms (e.g., scikit-learn and TensorFlow). It also analyzes the emotional data and incorporates the user's emotional state into the model via an emotion engine.

[0976] Input: Encrypted user personal and emotional data

[0977] Output: Integrated data including user model and emotional state

[0978] For example, the server normalizes large amounts of schedule data and voice data, analyzes them with machine learning algorithms, and generates a user model. Emotional data is also analyzed and reflected in the user model.

[0979] Step 4:

[0980] The server generates a digital clone based on the trained user model and emotion data. This digital clone is converted into binary format and sent to the device. The device stores the received digital clone locally and activates it when the user uses it.

[0981] Input: User model and emotion data

[0982] Output: Sending the binary digital clone to the device

[0983] As a concrete example, the server generates a digital clone based on the learned user model, encodes it as binary code, and sends it to the device, which then stores the received digital clone in local storage.

[0984] Step 5:

[0985] The user instructs the digital clone to perform an online task. For example, they give specific instructions such as, "Attend tomorrow's online meeting and take minutes." The device analyzes this instruction and sends it to the server. The server collects the necessary information and sends it back to the device. The digital clone then carries out the task and adjusts its responses and reports using an emotion engine.

[0986] Input: User instructions

[0987] Output: Task execution results and report to the user

[0988] As a concrete example, a user might say to the digital clone, "Take minutes of the meeting tomorrow at 9 o'clock." The device analyzes the instruction and sends it to the server. The server collects calendar information, and the digital clone automatically logs in to the meeting and creates minutes in real time using voice recognition. The emotion engine monitors emotions during the meeting and compiles appropriate minutes.

[0989] (Application example 2)

[0990] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0991] Conventional online task assistance systems do not take into account the user's emotional state, making it difficult to provide appropriate responses tailored to the user's actual needs and situation. Furthermore, they are unable to effectively utilize the user's preferences and past behavioral history, limiting the amount of suggestions and support they can provide. In particular, in the food delivery field, responses based on individual preferences, past behavioral history, and real-time emotional data are required to increase user satisfaction. The present invention aims to solve these problems.

[0992] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a device that collects personal data, a device that transmits the collected data to the server, a device that preprocesses the data received by the server and generates a user model using a machine learning algorithm, a device that generates a digital clone using the generated user model and transmits it to the terminal, a device that causes the digital clone to perform online tasks in accordance with the user's instructions, a device that recognizes emotional data in real time, and a device that analyzes the user's state using the recognized emotional data and suggests appropriate actions. This enables quick and accurate responses that take the user's emotional state into consideration, and in the food delivery field, it enables appropriate suggestions based on individual preferences and past behavioral history to increase user satisfaction.

[0993] "Personal data" refers to all data related to a user, such as the user's personal information, usage history, preference information, and allergy information.

[0994] A "server" is a computer system connected via a network that preprocesses data, runs machine learning algorithms, and generates user models and digital clones.

[0995] A "terminal" is a device used by a user, such as a computer or smartphone, that communicates with a server to receive a digital clone and realizes interaction with the user.

[0996] A "machine learning algorithm" is a computational method used to learn patterns and features in data and make predictions or classifications.

[0997] A "user model" is a digital profile that reflects a user's behavioral patterns and preferences, generated by machine learning algorithms.

[0998] A "digital clone" is a software agent that mimics a user's behavior and decisions, a virtual entity that performs online tasks according to the user's instructions.

[0999] "Emotion data" is data that indicates the emotional state of the user, obtained from the user's voice, facial expression, words, etc.

[1000] An "emotion engine" is a software module for recognizing and analyzing emotional data in real time.

[1001] "Online tasks" are all tasks that are performed using the Internet, and specifically include attending meetings, creating minutes, and managing schedules.

[1002] The "device for suggesting appropriate actions" is a device that recommends actions that are most suitable for the user's state based on collected emotional data and a user model.

[1003] "Food delivery" is a service that delivers food and drink to a location specified by the user.

[1004] The present invention relates to a system that uses a user's personal data and emotional data to generate a digital clone and assist with online tasks. The system collects personal data, transmits it to a server, performs data preprocessing, generates a user model using a machine learning algorithm, creates a digital clone, recognizes emotional data in real time, and suggests actions according to the user's state.

[1005] System configuration

[1006] The system includes the following components:

[1007] 1. Terminal devices for users to collect data

[1008] 2. A device where the server receives the data and generates a user model through preprocessing and machine learning algorithms.

[1009] 3. A device that uses a digital clone received by a terminal from a server to perform online tasks.

[1010] 4. A software module that enables the emotion engine to recognize and analyze emotion data in real time.

[1011] Explanation of program processing

[1012] The server receives personal and emotional data sent from the user's device. Data collection is performed using the smartphone's built-in microphone and camera, which capture voice and facial expression data. The received data is preprocessed on the server, and a user model is generated using a machine learning algorithm. This user model is a digital profile that reflects the user's past behavior and preferences. A digital clone is then generated based on this user model and sent to the user's device.

[1013] The emotion engine runs on the device and recognizes the user's real-time emotional state. The recognized emotional data is analyzed and suggestions appropriate to the user's state are made. For example, in a food delivery application, if a user is feeling stressed, the engine will suggest dishes that are suitable for relieving stress based on past preferences.

[1014] Usage example

[1015] Below are some specific examples of food delivery:

[1016] 1. The user types into the terminal, "I'm feeling stressed, so please give me some suggestions."

[1017] 2. The emotion engine detects the user's emotions in real time and recognizes "stress."

[1018] 3. The server analyzes the user's past order history and preferences based on the user model, and suggests dishes suitable for relieving stress.

[1019] 4. The device displays the suggested dishes to the user, and the user confirms the order.

[1020] 5. The server sends the order to the delivery service and manages the delivery status in real time.

[1021] Prompt Sentence Examples

[1022] An example of a prompt sentence for suggesting a dish to a user who is feeling stressed is as follows:

[1023] For example: "The user is currently feeling stressed. Their past ordering history shows that they like chocolate cake and comfort food to relieve stress. Please recommend appropriate dishes based on their current emotional state."

[1024] As described above, the system of the present invention can provide a fast and accurate response that takes into account the user's emotional state, which can increase user satisfaction, especially in the food delivery field. It can also provide high-level support using emotional data for other online tasks.

[1025] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1026] Step 1:

[1027] User collects data

[1028] Users use their smartphone's camera and microphone to collect personal and emotional data. Input data includes voice and facial expression data. This data is temporarily stored on the device. The collected voice and facial expression data is provided as output.

[1029] Step 2:

[1030] The device sends the data to the server

[1031] The device encrypts the collected personal data and emotion data and transmits them to the server via a secure protocol. The input data includes the collected voice and facial expression data. The output data includes the personal data and emotion data received by the server.

[1032] Step 3:

[1033] The server preprocesses the data

[1034] The server preprocesses the received data, removing noise and normalizing the data. The input data includes the server's personal data and emotion data sent from the device. After preprocessing, the cleaned data is stored on the server as the output.

[1035] Step 4:

[1036] The server generates the user model

[1037] The server uses the preprocessed data to generate a user model based on machine learning algorithms. The input data includes the preprocessed personal data and emotional data. The output is a user model, which is a digital profile that reflects the user's behavioral patterns and preferences.

[1038] Step 5:

[1039] The server creates a digital clone and sends it to the device.

[1040] The server creates a digital clone based on the generated user model and sends it to the terminal. The input data includes the user model. The output data includes the digital clone received by the terminal.

[1041] Step 6:

[1042] Emotion engine recognizes emotional data in real time

[1043] The emotion engine of the device recognizes the user's emotion data in real time. The input data includes real-time emotion data obtained from the user's voice and facial expressions. The output is the detected user's emotional state provided to the device.

[1044] Step 7:

[1045] The emotion engine analyzes the user's emotional state and makes appropriate suggestions to them.

[1046] The emotion engine on the device analyzes the recognized emotion data and the user model and makes suggestions appropriate to the user's state. For example, if a user is feeling stressed, it will suggest dishes that are good for relieving stress. The input data includes the detected emotional state and the user model. The output is a display of the suggestions made to the user.

[1047] Step 8:

[1048] The user confirms the order

[1049] The user confirms the order based on the suggestions displayed on the terminal. The input data includes the suggestions displayed on the terminal. The output is the confirmed order sent to the server.

[1050] Step 9:

[1051] The server processes the order and sends it to the delivery service

[1052] The server processes the user's confirmed order and sends it to the delivery service. The input data includes the user's confirmed order. The output data includes the order details received by the delivery service.

[1053] Step 10:

[1054] The server manages the delivery status

[1055] The server tracks the delivery status from the delivery service in real time and provides the information to the user. The input data includes delivery progress information from the delivery service. The output data includes delivery progress information provided to the user.

[1056] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1057] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1058] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1059] [Third embodiment]

[1060] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1061] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1062] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1063] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1064] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1065] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1066] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1067] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1068] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1069] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1070] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1071] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1072] The present invention relates to a system for creating a digital clone using personal data of a user and assisting with online tasks. Specific embodiments of the system are described below.

[1073] Digital Clone System Overview

[1074] The system of the present invention is composed of three main components: a "server," a "terminal," and a "user." Users interact with the system using dedicated applications. Data collection, processing, cloning, and task assistance are performed, and each step to achieve this is described in detail below.

[1075] User Data Collection

[1076] Users provide their own data, including email history, schedules, voice data, photos, internet history, etc. Users select this data through a dedicated application and upload it to the system.

[1077] The device temporarily stores this data and transmits it to a server in encrypted form using a secure protocol.

[1078] Data Preprocessing and Training

[1079] The server preprocesses the received data, which includes cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences and generate a user model. This model serves as the basis for creating a digital clone of the user.

[1080] Creating a digital clone

[1081] The server generates a digital clone based on the learned user model. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions.

[1082] The device stores the received digital clone locally and activates it when the user wants to use it.

[1083] Online task assistance

[1084] The user instructs the digital clone to perform online tasks, such as "attend tomorrow's online meeting and take minutes."

[1085] The device analyzes this instruction and requests the necessary information from the server, which then collects meeting information from the user's schedule management system and sends the details to the device.

[1086] The digital clone automatically logs in at the designated meeting time, performs real-time voice recognition, and converts the conversation into text. When the meeting ends, a summary of the meeting minutes is sent to the user via the device.

[1087] Specific examples

[1088] Example 1: Creating minutes for online meetings

[1089] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1090] 2. The device sends this instruction to the server.

[1091] 3. The server collects the meeting details (participant list, agenda, etc.) and sends them to the terminal.

[1092] 4. A digital clone logs in at the meeting time and recognizes and transcribes the conversation in real time.

[1093] 5. The digital clone compiles minutes and sends them to the user via the terminal after the meeting ends.

[1094] Example 2: Task Management

[1095] 1. The user asks, "Organize this week's tasks and set reminders."

[1096] 2. The device sends the user's schedule to the server.

[1097] 3. The server analyzes the user's schedule and identifies high-priority tasks.

[1098] 4. The server lists the identified tasks, sets reminders and sends them to the device.

[1099] 5. The device notifies the user of the reminder.

[1100] How to use

[1101] To use the system of the present invention, a user installs a dedicated application and registers as a user. Next, the user selects and provides the data they want to collect, and begins creating a digital clone. Once the digital clone is created, the user can instruct the digital clone to perform various online tasks through the dedicated application, improving work efficiency.

[1102] In this way, by implementing the digital clone system of the present invention, users can efficiently manage and carry out their own work, thereby reducing their workload.

[1103] The processing flow will be explained below.

[1104] Digital clone system processing flow

[1105] Data Collection Phase

[1106] Step 1:

[1107] The user launches the dedicated application and selects the data they want to collect (emails, schedules, audio data, photos, internet history, etc.).

[1108] Step 2:

[1109] The device temporarily stores selected data, for example, email history in a temporary folder in local storage.

[1110] Step 3:

[1111] The device encrypts the stored data using a secure communication protocol (e.g., SSL / TLS) and sends it to the server.

[1112] Data preprocessing and training phase

[1113] Step 4:

[1114] The server preprocesses the data it receives. Specifically, it cleans and filters the data to remove unnecessary noise. For example, it removes spam and advertising emails from email data.

[1115] Step 5:

[1116] The server feeds the preprocessed data into machine learning algorithms to generate a user model, a process that learns and models the user's behavioral patterns and preferences.

[1117] Digital Clone Generation Phase

[1118] Step 6:

[1119] The server generates a digital clone based on the learned user model. This digital clone is an agent that simulates the user's actions and decisions.

[1120] Step 7:

[1121] The server converts the generated digital clone into binary format and transmits it to the terminal.

[1122] Step 8:

[1123] The device stores the received digital clone locally and launches it when the user needs it.

[1124] Online task assistance phase

[1125] Step 9:

[1126] The user gives the digital clone tasks through a dedicated application, such as "attend tomorrow's online meeting and take minutes."

[1127] Step 10:

[1128] The terminal analyzes the user's instructions and sends a request to the server to obtain the required information.

[1129] Step 11:

[1130] The server collects the user's schedule and meeting information and sends it to the device, for example extracting meeting details from the user's calendar.

[1131] Step 12:

[1132] Based on the information acquired by the device, a digital clone is activated and allowed to participate in the meeting. The digital clone converts speech into text in real time and creates minutes.

[1133] Step 13:

[1134] After the meeting, the digital clone compiles minutes and sends them to the user via the device.

[1135] Step 14:

[1136] The user reviews the minutes they receive and gives the digital clone further instructions, such as "Share the minutes with the team."

[1137] Specific examples

[1138] Example 1: Process flow for creating minutes of an online meeting

[1139] Step 1:

[1140] The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1141] Step 2:

[1142] The terminal analyzes the instructions and sends them to the server.

[1143] Step 3:

[1144] The server collects the details of the meeting (participant list, agenda, etc.) and sends them to the terminal.

[1145] Step 4:

[1146] The digital clone automatically logs in at meeting time and recognizes voice and transcribes the conversation in real time.

[1147] Step 5:

[1148] The digital clone compiles minutes and sends them to the user via the terminal after the meeting ends.

[1149] Example 2: Task management process flow

[1150] Step 1:

[1151] The user says, "Organize this week's tasks and set reminders."

[1152] Step 2:

[1153] The terminal transmits the user's schedule information to the server.

[1154] Step 3:

[1155] The server analyzes the schedule and identifies important and urgent tasks.

[1156] Step 4:

[1157] The server lists the identified tasks and sends them to the terminal.

[1158] Step 5:

[1159] The device notifies the user of the reminder.

[1160] These detailed processing steps allow the digital clone system of the present invention to efficiently perform tasks according to the user's instructions, thereby reducing the burden on the user.

[1161] Example 1

[1162] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1163] In today's world, with the increase in online tasks, it is becoming increasingly difficult for individual users to efficiently manage their time and complete tasks. Furthermore, taking minutes of online meetings and managing tasks is time-consuming and accuracy is an issue. This has led users to seek ways to improve the efficiency of their work.

[1164] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1165] In this invention, the server includes means for collecting personal data, means for transmitting the collected data to the server using an encryption protocol, means for the server to preprocess the received data and generate a user model using a machine learning algorithm, means for the server to create a digital clone using the generated user model and transmit it to the terminal, means for the digital clone to analyze the user's instructions using natural language processing and execute online tasks, and means for the digital clone to notify the user of the results after the task is executed. This enables users to efficiently manage and execute online tasks, significantly improving work efficiency.

[1166] "Personal data" refers to any form of information related to a user (email history, schedule, audio data, photos, internet history, etc.).

[1167] "Cryptographic protocol" refers to the technology (e.g., TLS) used to ensure the secure communication of data.

[1168] A "server" refers to a computer system that performs core computations, such as data preprocessing, generating machine learning models, and creating digital clones.

[1169] A "machine learning algorithm" refers to a computational method that learns patterns from data and generates and updates a user model.

[1170] A "user model" refers to a data structure that represents a user's behavioral patterns and preferences, generated by a machine learning algorithm based on collected personal data.

[1171] A "digital clone" refers to a software agent that is generated based on a user model and mimics the user's behavior and decisions.

[1172] "Terminal" refers to the device (e.g., smartphone, tablet, PC) through which a user interacts with their digital clone.

[1173] "Natural language processing" refers to the technology that a digital clone uses to understand and interpret user instructions.

[1174] "Online tasks" refer to Internet tasks that a user instructs a digital clone to perform (e.g., participating in an online meeting, task management).

[1175] "Speech recognition technology" refers to technology that converts speech into text in real time.

[1176] "Minutes" refers to a document that records the contents of the conversation that took place during an online meeting as text.

[1177] "Reminder" refers to a function that notifies the user according to a schedule.

[1178] "Information collection system" refers to external APIs and tools for collecting user schedules and other necessary information.

[1179] The present invention relates to a system for creating a digital clone using personal data of a user and assisting with online tasks. Specific embodiments of the system are described below.

[1180] User Data Collection

[1181] Users use a dedicated application to provide their own data, including email history, schedules, voice data, photos, internet history, etc. The user selects this data and uploads it to the system.

[1182] The device temporarily stores this data and transmits it to the server using an encryption protocol such as TLS (Transport Layer Security).

[1183] Data Preprocessing and Training

[1184] The server preprocesses the received data, including cleaning the data (e.g., imputing missing values ​​and removing outliers), filtering, and normalization. This preprocessing is performed using Python's Pandas and Numpy libraries.

[1185] The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences, generating a user model using libraries like TensorFlow and PyTorch. This user model serves as the basis for creating a digital clone of the user.

[1186] Creating a digital clone

[1187] The server generates a digital clone based on the learned user model. This digital clone is a software agent generated using Python scripts that mimics the user's behavior and decisions. The generated digital clone is converted into a binary format and sent to the device.

[1188] The terminal stores the received digital clone in local storage, using SQLite or JSON format. When the user wants to use it, the terminal launches the digital clone, possibly using a Python script or a Docker container.

[1189] Online task assistance

[1190] The user instructs the digital clone to perform online tasks. These instructions are entered as prompt sentences through the interface of a dedicated application. For example, the user might instruct the clone to "attend tomorrow's online meeting and take minutes."

[1191] The device parses this instruction using natural language processing (NLP) techniques (e.g., spaCy or NLTK libraries). The device then requests the necessary information from the server. For example, it could retrieve meeting information from the Google Calendar API.

[1192] The server collects meeting information from the user's scheduler and sends the details to the device. The digital clone then automatically logs in at the designated meeting time and performs real-time speech recognition and converts the conversation into text, using, for example, the Google Cloud Speech-to-Text API. Once the meeting is over, a summary of the meeting minutes is reported to the user via the device.

[1193] Specific examples

[1194] Example 1: Creating minutes for online meetings

[1195] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1196] 2. The device analyzes this instruction and requests the necessary information from the server.

[1197] 3. The server uses the Google Calendar API to collect the meeting details and send them to the device.

[1198] 4. At the meeting time, the digital clone logs in using Selenium and converts the audio to text using the Google Cloud Speech-to-Text API.

[1199] 5. The digital clone compiles the minutes in Markdown format and notifies the user through a dedicated application.

[1200] Example 2: Task Management

[1201] 1. The user asks, "Organize this week's tasks and set reminders."

[1202] 2. The terminal sends the user's schedule information to the server.

[1203] 3. The server uses Natural Language Processing (NLP) technology to analyze the schedule and identify high-priority tasks.

[1204] 4. The server lists the identified tasks, sets reminders using the Google Calendar API, and sends them to the device.

[1205] 5. The device notifies the user of the reminder.

[1206] Prompt Sentence Examples

[1207] "Please take the minutes of the meeting tomorrow at 9 o'clock."

[1208] "Organize my tasks for this week and set reminders."

[1209] By following this specific process flow, users can efficiently manage and complete online tasks.

[1210] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1211] Step 1:

[1212] The user launches a dedicated application and provides their personal data (e.g., email history, schedule, audio data, photos, internet history).

[1213] Input: Personal Data

[1214] Output: Temporarily saved dataset

[1215] Specific operations: Select the type of data to provide through the dedicated application interface and press the button to upload the data.

[1216] Step 2:

[1217] The device temporarily stores the data uploaded by the user and sends it to the server using an encryption protocol such as TLS (Transport Layer Security).

[1218] Input: Temporarily saved dataset

[1219] Output: Data in encrypted form

[1220] Specific operation: The data is temporarily stored in a local database (e.g., SQLite) on the device, and then encrypted using the OpenSSL library. After encryption, the data is sent to the server.

[1221] Step 3:

[1222] The server stores the received data in a cloud database (e.g., Amazon RDS) and performs preprocessing.

[1223] Input: Encrypted data

[1224] Output: Preprocessed dataset

[1225] Specific operations: Decrypt the data and use the Pandas library to clean (fill in missing values, remove outliers), filter, and normalize the data.

[1226] Step 4:

[1227] The server inputs the preprocessed data into a machine learning algorithm (e.g., TensorFlow, PyTorch) to generate a user model.

[1228] Input: Preprocessed dataset

[1229] Output: User model

[1230] Specific operation: Data is fed into a machine learning algorithm to learn behavioral patterns and preferences. As a result, a user model is generated.

[1231] Step 5:

[1232] The server generates a digital clone based on the user model, converts it into binary format, and transmits it to the terminal.

[1233] Input: User model

[1234] Output: Digital clone in binary format

[1235] Specific operation: The user model is embodied as a digital clone using a Python script, then converted into binary format. The converted digital clone is sent to the terminal.

[1236] Step 6:

[1237] The terminal stores the received digital clone in local storage (e.g., SQLite, JSON format) and activates the digital clone as needed.

[1238] Input: Digital clone in binary format

[1239] Output: Activated digital clone

[1240] Specific operation: A digital clone is stored in local storage and launched using a Python script or Docker container when the user uses it.

[1241] Step 7:

[1242] The user instructs the digital clone on tasks through a dedicated application.

[1243] Input: Prompt sentence (e.g., "Please attend tomorrow's online meeting and take minutes.")

[1244] Output: Parsed task instructions

[1245] Specific operations: Enter the prompt text through the dedicated application interface and press the send button.

[1246] Step 8:

[1247] The device analyzes the user's instructions using natural language processing (NLP) technology (e.g., spaCy, NLTK) and requests the necessary information from the server.

[1248] Input: prompt statement

[1249] Output: Task execution information request

[1250] Specific operations: Analyzes the text of the user's instructions using NLP technology, interprets the task content, and sends a request to the server for the appropriate information.

[1251] Step 9:

[1252] The server collects the necessary information from the user's schedule management system (e.g., Google Calendar API) and sends it to the device.

[1253] Input: Task execution information request

[1254] Output: Collected schedule information

[1255] Specific operation: Uses API to obtain user schedule information and sends it to the device.

[1256] Step 10:

[1257] The digital clone performs online tasks (e.g., participating in an online meeting) and notifies the user of the results.

[1258] Input: Collected schedule information

[1259] Output: Task results (e.g. meeting minutes)

[1260] Specific operations: Log in at the specified meeting time (e.g., using Selenium), convert the audio into text using the Google Cloud Speech-to-Text API, summarize the task results, and notify the user.

[1261] Prompt Sentence Examples

[1262] "Please take the minutes of the meeting tomorrow at 9 o'clock."

[1263] "Organize my tasks for this week and set reminders."

[1264] The above is the specific processing flow of this system.

[1265] (Application example 1)

[1266] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1267] While existing technologies exist that generate digital clones based on a user's personal data and assist with online tasks, they lack the means to analyze a customer's purchase history and preferences in physical stores and recommend appropriate products. As a result, improvements to customer service and the shopping experience in physical stores have not been fully realized. The present invention aims to solve this problem and improve the customer experience in physical stores.

[1268] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1269] In this invention, the server includes a means for acquiring personal data, a means for transmitting the acquired data to the server, a means for the server to preprocess the received data and generate a user model using a machine learning algorithm, a means for creating a digital clone using the generated user model and transmitting it to the terminal, a means for the digital clone to perform online tasks according to the user's instructions, and a means for the digital clone to analyze the customer's purchase history and preferences using smart glasses in a physical store and recommend appropriate products. This enables the digital clone to immediately respond to customer needs even in a physical store and recommend optimal products.

[1270] "Personal data" refers to data that includes a user's behavioral history, purchasing history, preferences, and other personal information.

[1271] A "server" is a computer system that receives user data, pre-processes it, generates models using machine learning, and creates and transmits digital clones.

[1272] A "terminal" is a device that receives a digital clone sent from a server and allows a user to interact with the digital clone.

[1273] A "machine learning algorithm" is a technology that analyzes user data and learns user behavior patterns and preferences.

[1274] A "user model" is a data model that reflects a user's behavioral patterns and preferences, generated using a machine learning algorithm.

[1275] A "digital clone" is a software agent that mimics a user's behavior and decisions based on a generated user model.

[1276] An "online task" is a task performed over the Internet and executed by the digital clone based on the user's instructions.

[1277] A "physical store" is a physically existing sales store that is a place that users (customers) visit in person.

[1278] "Smart glasses" are wearable devices that have augmented reality and information display functions and are used by store clerks in physical stores when providing customer service.

[1279] "Purchase history" is a record of products that a customer has purchased in the past.

[1280] "Preferences" refer to a user's tendency to prefer a particular type or style of product.

[1281] In the context of "Levidence," a "designated document" is a document submitted in the course of a trial or investigation that is referenced based on specific requirements.

[1282] The present invention is a system for collecting personal data of users and generating digital clones, and it is particularly intended to provide personalized product recommendations to customers using smart glasses in brick-and-mortar stores. Specific embodiments of the system are described below.

[1283] System Overview

[1284] The system of the present invention consists of a server, a terminal, and a user. The user interacts with the system using smart glasses, and data collection, processing, digital cloning, and product recommendations are performed. This procedure is described in detail below.

[1285] Retrieving User Data

[1286] Users provide their own data, including purchase history, preferences, and other behavioral patterns. They select this data through a dedicated application and upload it to the system. The device temporarily stores this data and transmits it to the server in encrypted form using a secure protocol.

[1287] Data preprocessing and user model generation

[1288] The server preprocesses the received data, which includes cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms (e.g., the scikit-learn library) to learn the user's behavioral patterns and preferences and generate a user model.

[1289] Creating and transmitting digital clones

[1290] The server generates a digital clone based on the learned user model. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions. The device stores the received digital clone locally and activates it when the user uses it.

[1291] Product recommendations in physical stores

[1292] A user wears smart glasses and performs the task of recommending products to a customer. For example, a customer may request, "Please show me the latest new products." The device sends this instruction to the server, which analyzes the customer's purchase history and preferences. Based on the analysis results, the device sends a recommendation list of appropriate new products. The user can then suggest the best products to the customer based on the information displayed on the smart glasses' display.

[1293] Specific examples

[1294] Usage Scenarios

[1295] 1. Customer A asks the store clerk to show him the latest new products.

[1296] 2. The device sends this instruction to the server.

[1297] 3. The server analyzes customer A's purchase history and preferences, generates a list of recommended products, and sends it to the terminal.

[1298] 4. The user (store clerk) introduces the recommended product to customer A through the smart glasses.

[1299] Prompt Sentence Examples

[1300] "Customer ID: customer123. Looking at their recent purchase history, they have purchased a bag, shoes, and a jacket. New items have arrived. What products would you recommend for this customer?"

[1301] Hardware and software used

[1302] Hardware: Servers (e.g., Amazon Web Services, Google Cloud Platform)

[1303] Software: Python libraries (e.g., requests, scikit-learn)

[1304] This will enable the digital clone to instantly respond to customer needs in physical stores and recommend the most suitable products.

[1305] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1306] Step 1:

[1307] User provides data

[1308] Users select data such as their purchase history, preferences, and behavioral patterns through a dedicated application and upload that data to the device. INPUT is the user's personal information and history data, and OUTPUT is encrypted data stored on the device.

[1309] Step 2:

[1310] The device sends data to the server

[1311] The device temporarily stores the acquired data and sends it to the server in encrypted form using a secure protocol (e.g., HTTPS). INPUT is the encrypted data stored on the device, and OUTPUT is the encrypted data sent to the server.

[1312] Step 3:

[1313] The server preprocesses the data

[1314] The server performs pre-processing on the received data, such as cleaning, filtering, and normalization. Unwanted noise is removed and the data is in a format that can be analyzed. INPUT is the encrypted data sent to the server, and OUTPUT is the pre-processed data.

[1315] Step 4:

[1316] The server generates a user model using a machine learning algorithm

[1317] The server uses the preprocessed data to apply machine learning algorithms (e.g., scikit-learn) to learn the user's behavioral patterns and preferences, which results in a user model. The input is the preprocessed data, and the output is the user model.

[1318] Step 5:

[1319] The server uses the user model to create a digital clone

[1320] The server creates a digital clone based on the generated user model. The created digital clone is converted into binary format and sent to the terminal. The input is the user model, and the output is the binary format digital clone.

[1321] Step 6:

[1322] The device stores the digital clone locally and activates it when the user wants to use it.

[1323] The terminal stores the received digital clone locally and activates it as needed. This allows the user to receive services from the digital clone. INPUT is the digital clone sent from the server, and OUTPUT is the digital clone stored in the terminal.

[1324] Step 7:

[1325] Users wear smart glasses to serve customers

[1326] The user (store clerk) wears smart glasses and receives requests from customers (e.g., "Please show me the latest new products"). INPUT is the customer's request, and OUTPUT is the instruction sent to the terminal.

[1327] Step 8:

[1328] The terminal sends the customer's request to the server

[1329] The terminal analyzes the customer's request and sends the information to the server. INPUT is the customer's request, and OUTPUT is the request to the server.

[1330] Step 9:

[1331] The server analyzes the customer's purchase history and preferences to recommend appropriate products.

[1332] The server analyzes the customer's purchase history and preferences based on the received request. This generates an appropriate product recommendation list and sends it to the terminal. INPUT is the request to the server and customer data, and OUTPUT is the list of recommended products.

[1333] Step 10:

[1334] Users can suggest products to customers through smart glasses

[1335] The user (store clerk) recommends the best products to the customer based on the product recommendation list displayed on the smart glasses display. The input is the list of recommended products, and the output is the product proposal to the customer.

[1336] This allows the digital clone to instantly respond to customer needs in a physical store and recommend the most suitable products.

[1337] "Customer ID: customer123. Looking at their recent purchase history, they have purchased a bag, shoes, and a jacket. New items have arrived. What products would you recommend for this customer?"

[1338] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1339] The present invention relates to a system for creating a digital clone using personal data and emotional data of a user and assisting the user in online tasks. Specific embodiments of the system are described below.

[1340] Digital Clone System Overview

[1341] The system of the present invention consists of four main components: a "server," a "terminal," a "user," and an "emotion engine." The user interacts with the system using a dedicated application, which collects data, processes it, creates clones, and assists with tasks. The emotion engine also recognizes the user's emotions in real time and utilizes this information throughout the system.

[1342] User and sentiment data collection

[1343] The user selects the data they want to collect (email, schedule, voice data, photos, internet history, etc.) through a dedicated application, and simultaneously collects emotional data (voice and facial expression data). Emotional data is acquired through sensors such as a microphone and camera.

[1344] The device temporarily stores this data and transmits it to a server in encrypted form using a secure protocol.

[1345] Data Preprocessing and Training

[1346] The server preprocesses the received data. This involves cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences and generate a user model. This model serves as the basis for creating a digital clone of the user.

[1347] The server similarly processes the emotion data obtained through the emotion engine and incorporates the user's emotional state into the model.

[1348] Creating a digital clone

[1349] The server generates a digital clone based on the trained user model and emotional data. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions, and responds taking into account the user's emotional state.

[1350] The device stores the received digital clone locally and activates it when the user wants to use it.

[1351] Online task assistance

[1352] The user instructs the digital clone to perform online tasks, such as "attend tomorrow's online meeting and take minutes."

[1353] The device analyzes this instruction and requests the necessary information from the server, which then collects meeting information from the user's schedule management system and sends the details to the device.

[1354] The digital clone automatically logs in at the designated meeting time, performs real-time speech recognition, and converts the conversation into text. When the meeting ends, a summary of the meeting minutes is sent to the user via their device. In addition, an emotion engine monitors the user's emotional state during the meeting and creates appropriate responses and meeting minutes based on that information.

[1355] Specific examples

[1356] Example 1: Creating minutes for online meetings

[1357] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1358] 2. The device sends this instruction to the server.

[1359] 3. The server collects the meeting details (participant list, agenda, etc.) and sends them to the terminal.

[1360] 4. The digital clone automatically logs in at the meeting time, recognizes the conversation in real time, and transcribes it into text. The emotion engine simultaneously monitors the person's emotional state.

[1361] 5. The digital clone compiles minutes of the meeting, taking into account emotional data, and sends them to the user via their device after the meeting ends.

[1362] Example 2: Task Management

[1363] 1. The user asks, "Organize this week's tasks and set reminders."

[1364] 2. The terminal sends the user's schedule information to the server.

[1365] 3. The server analyzes the schedule and identifies important and urgent tasks, taking into account emotional data.

[1366] 4. The server lists the identified tasks and sends them to the device.

[1367] 5. The device notifies the user of the reminder. The emotion engine provides feedback on the user's emotional state and adjusts the wording of the reminder.

[1368] How to use

[1369] To use the system of the present invention, a user installs a dedicated application and registers as a user. Next, the user selects and provides the data they want to collect, and begins creating a digital clone. Once the digital clone is created, the user can instruct the digital clone to perform various online tasks through the dedicated application, improving work efficiency.

[1370] Furthermore, the introduction of an emotion engine enables the digital clone to respond and act in a way that takes into account the user's emotional state, providing more appropriate and effective support.

[1371] In this way, by implementing the digital clone system of the present invention, users can efficiently manage and carry out their own work, reducing their workload. The introduction of an emotion engine will provide even more advanced support.

[1372] The processing flow will be explained below.

[1373] Digital clone system processing flow

[1374] Data Collection Phase

[1375] Step 1:

[1376] The user launches the dedicated application and selects the data they want to collect (emails, schedules, audio data, photos, internet history, etc.).

[1377] Step 2:

[1378] The device temporarily stores the selected data, while simultaneously collecting the user's voice and facial expression data using the microphone and camera.

[1379] Step 3:

[1380] The device analyzes the emotional state using voice and facial expression data and generates emotional data.

[1381] Step 4:

[1382] The personal data and emotional data collected by the device are sent to a server using a secure protocol (e.g., encryption using SSL / TLS).

[1383] Data preprocessing and training phase

[1384] Step 5:

[1385] The server preprocesses the received personal and emotional data, specifically cleaning, filtering, and denoising the data.

[1386] Step 6:

[1387] The server then feeds the pre-processed data into machine learning algorithms to generate a user model that reflects the user's behavioral patterns and emotional state.

[1388] Digital Clone Generation Phase

[1389] Step 7:

[1390] The server generates a digital clone based on the learned user model and emotional data. The digital clone simulates the user's behavior and decisions and responds according to the user's emotions.

[1391] Step 8:

[1392] The server converts the generated digital clone into binary format and transmits it to the terminal.

[1393] Step 9:

[1394] The device stores the received digital clone locally and activates it when the user gives the command.

[1395] Online task assistance phase

[1396] Step 10:

[1397] The user issues tasks to the digital clone through a dedicated application, such as "attend tomorrow's online meeting and take minutes."

[1398] Step 11:

[1399] The terminal analyzes the user's instructions and sends a request to the server to obtain the required information.

[1400] Step 12:

[1401] The server collects the user's schedule and meeting information and sends it to the terminal.

[1402] Step 13:

[1403] The device activates a digital clone based on the information it acquires and automatically logs in at the specified meeting time.

[1404] Step 14:

[1405] The digital clone converts speech to text in real time during meetings and creates meeting minutes, while the emotion engine monitors the user's emotional state and suggests appropriate responses.

[1406] Step 15:

[1407] After the meeting, the digital clone will compile minutes and report them to the user via their device, taking into account emotional data.

[1408] Specific examples

[1409] Example 1: Process flow for creating minutes of an online meeting

[1410] Step 1:

[1411] The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1412] Step 2:

[1413] The terminal sends the instruction to the server.

[1414] Step 3:

[1415] The server collects the details of the meeting (participant list, agenda, etc.) and sends them to the terminal.

[1416] Step 4:

[1417] A digital clone logs in at meeting time, converts speech to text in real time, and creates minutes. An emotion engine monitors the user's emotional state during the meeting.

[1418] Step 5:

[1419] The digital clone will compile minutes and report them to the user via their device after the meeting, taking into account emotional data.

[1420] Example 2: Task management process flow

[1421] Step 1:

[1422] The user says, "Organize this week's tasks and set reminders."

[1423] Step 2:

[1424] The terminal transmits the user's schedule information and emotion data to the server.

[1425] Step 3:

[1426] The server analyzes schedule and emotion data to identify important and urgent tasks.

[1427] Step 4:

[1428] The server lists the identified tasks, generates reminders based on the user's emotional state, and sends them to the device.

[1429] Step 5:

[1430] The device notifies the user of the reminder, and the emotion engine monitors the user's emotional state and adjusts the wording of the reminder accordingly.

[1431] These detailed processing steps enable the digital clone system of the present invention to efficiently perform tasks according to the user's instructions and provide appropriate support that takes into account the user's emotional state.

[1432] Example 2

[1433] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1434] While conventional digital assistant systems could assist users with online tasks using their personal data, they lacked the ability to adjust responses and behaviors based on the user's emotional state. This resulted in problems such as stress and insufficient support. Furthermore, they lacked the ability to respond in real time for specific online tasks, such as taking meeting minutes. Furthermore, prioritizing tasks and setting reminders did not take the user's emotional state into account, which meant they did not fully meet the user's needs.

[1435] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for collecting personal data and emotional data; means for encrypting the collected data and transmitting it to the server; means for the server to preprocess the received data and generate a user model using a machine learning algorithm; means for creating a digital clone based on the generated user model and emotional data and transmitting it to the terminal; and means for the digital clone to perform online tasks according to the user's instructions and adjust responses and report content using an emotion engine. This makes it possible to provide more personalized assistance by utilizing the user's personal data and emotional data, and to realize responses and actions appropriate to the user's emotional state. Furthermore, appropriate services that take the user's emotions into consideration can be provided when efficiently completing specific online tasks and prioritizing tasks.

[1436] "Personal data" refers to all information about the user, including emails, schedules, audio data, photos, internet history, and so on.

[1437] "Emotion data" refers to data that reflects the user's emotional state, and includes information such as voice and facial expressions acquired using a microphone or camera.

[1438] "Server" refers to the computer system that collects, preprocesses, analyzes, generates models, and creates digital clones of data.

[1439] "Terminal" refers to a device used by a user to operate the system, including smartphones and personal computers.

[1440] "Encryption" refers to the process of converting data into a special format for secure communication.

[1441] "Preprocessing" refers to processes such as data cleaning, filtering, and noise removal that transform raw data into a format suitable for machine learning algorithms.

[1442] A "machine learning algorithm" refers to a computational method that learns patterns from data and uses them to make predictions and classifications.

[1443] "User model" refers to a digital representation of a user's behavior and preferences that is generated based on collected personal and emotional data.

[1444] A "digital clone" refers to a software agent that is created based on a user model and is used to carry out user instructions and tasks.

[1445] "Emotion engine" refers to a function that analyzes user emotional data and utilizes that information throughout the system.

[1446] "Online tasks" refer to work or tasks carried out over the Internet, including online meetings, taking minutes, and schedule management.

[1447] The present invention relates to a system that uses a user's personal data and emotional data to create a digital clone and assist with online tasks. This system mainly consists of four components: a "server," a "terminal," a "user," and an "emotion engine." The following describes the specific means and process for implementing the present invention.

[1448] 1. Initial Setup and Data Collection

[1449] Users install the dedicated application and set up their account. Next, they select the type of data to be collected (email, schedule, voice data, photos, internet history, etc.) and set the corresponding access permissions for the application. Emotional data is collected using the microphone and camera.

[1450] For example, a user launches a dedicated smartphone app and registers the day's calendar information and schedule. At the same time, the camera captures their facial expressions and the microphone records their mood for the day.

[1451] 2. Data Preprocessing and Encryption

[1452] The device temporarily stores the collected data and performs preprocessing using the Python Pandas library to clean and denoise the data. After preprocessing, the data is encrypted using AES encryption and sent to the server using a secure protocol (e.g., HTTPS).

[1453] 3. Training and Model Generation

[1454] The server analyzes the received data and generates a user model using machine learning algorithms (e.g., scikit-learn or TensorFlow). Python libraries such as Pandas and NumPy are used for data cleaning and normalization. Emotion data is also processed through an emotion engine, which incorporates the user's emotional state into the model.

[1455] 4. Creating a Digital Clone

[1456] The server generates a digital clone based on the trained user model and emotion data. This digital clone is converted into binary format and sent to the device. The device stores the received digital clone locally and activates it when the user uses it.

[1457] 5. Assistance and execution of tasks

[1458] The user gives the digital clone instructions, such as "Take minutes of tomorrow's online meeting." The device analyzes these instructions, sends them to the server, and retrieves the necessary information. The digital clone then participates in the online meeting, converts speech to text in real time, and generates minutes. It uses an emotion engine to adjust responses and reports based on the user's emotional state.

[1459] Specific examples

[1460] Example 1: Creating minutes for online meetings

[1461] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1462] 2. The device sends this instruction to the server.

[1463] 3. The server collects the meeting details (participant list, agenda) and sends them to the terminal.

[1464] 4. The digital clone automatically logs in at the meeting time and converts the conversation into text using voice recognition. The emotion engine monitors the user's emotional state during the meeting and finalizes the minutes.

[1465] 5. The digital clone compiles the minutes and sends them to the user via the terminal.

[1466] Example 2: Task Management

[1467] 1. The user asks, "Organize this week's tasks and set reminders."

[1468] 2. The terminal sends the user's schedule information to the server.

[1469] 3. The server analyzes the schedule data and identifies important and urgent tasks, taking into account emotional data.

[1470] 4. The server lists the identified tasks and sends them to the device.

[1471] 5. The device notifies the user of the reminder. The emotion engine provides feedback on the user's emotional state and adjusts the wording of the reminder.

[1472] Prompt Sentence Examples

[1473] "Describe a system where a user provides data using a dedicated app to generate a digital clone along with emotional data."

[1474] In this way, the digital clone system of the present invention can significantly improve users' work efficiency by analyzing users' personal data and emotional data in real time and providing optimal online task support. The introduction of an emotion engine enables responses and actions that take the user's emotional state into account, providing more effective and personalized support.

[1475] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1476] Step 1:

[1477] Users install the dedicated application and set up an account. After launching the application, they select the type of data to be collected (email, schedule, voice data, photos, internet history, etc.) and set the corresponding access permissions for the application. Emotional data is also collected using the microphone and camera.

[1478] Input: User's personal and emotional data

[1479] Output: Temporarily save collected data to the device

[1480] For example, a user launches a dedicated smartphone app and allows access to calendar information and the microphone. The app then captures facial expressions with the camera and records the user's mood for the day with the microphone.

[1481] Step 2:

[1482] The terminal temporarily stores the collected data. Then, the data is cleaned to remove unnecessary noise and invalid entries. Specifically, the data is cleaned using the Python Pandas library. The data is also AES encrypted and sent to the server using a secure protocol (e.g., HTTPS).

[1483] Input: User's personal and emotional data

[1484] Output: Preprocessed encrypted data sent to the server

[1485] As a specific example, the terminal receives "schedule data" and "audio data," removes noise, applies AES encryption, and sends it to the server via HTTPS.

[1486] Step 3:

[1487] The server analyzes the received data and further cleans and normalizes it. Specifically, it processes the data using Pandas and NumPy. It then generates a user model using machine learning algorithms (e.g., scikit-learn and TensorFlow). It also analyzes the emotional data and incorporates the user's emotional state into the model via an emotion engine.

[1488] Input: Encrypted user personal and emotional data

[1489] Output: Integrated data including user model and emotional state

[1490] For example, the server normalizes large amounts of schedule data and voice data, analyzes them with machine learning algorithms, and generates a user model. Emotional data is also analyzed and reflected in the user model.

[1491] Step 4:

[1492] The server generates a digital clone based on the trained user model and emotion data. This digital clone is converted into binary format and sent to the device. The device stores the received digital clone locally and activates it when the user uses it.

[1493] Input: User model and emotion data

[1494] Output: Sending the binary digital clone to the device

[1495] As a concrete example, the server generates a digital clone based on the learned user model, encodes it as binary code, and sends it to the device, which then stores the received digital clone in local storage.

[1496] Step 5:

[1497] The user instructs the digital clone to perform an online task. For example, they give specific instructions such as, "Attend tomorrow's online meeting and take minutes." The device analyzes this instruction and sends it to the server. The server collects the necessary information and sends it back to the device. The digital clone then carries out the task and adjusts its responses and reports using an emotion engine.

[1498] Input: User instructions

[1499] Output: Task execution results and report to the user

[1500] As a concrete example, a user might say to the digital clone, "Take minutes of the meeting tomorrow at 9 o'clock." The device analyzes the instruction and sends it to the server. The server collects calendar information, and the digital clone automatically logs in to the meeting and creates minutes in real time using voice recognition. The emotion engine monitors emotions during the meeting and compiles appropriate minutes.

[1501] (Application example 2)

[1502] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1503] Conventional online task assistance systems do not take into account the user's emotional state, making it difficult to provide appropriate responses tailored to the user's actual needs and situation. Furthermore, they are unable to effectively utilize the user's preferences and past behavioral history, limiting the amount of suggestions and support they can provide. In particular, in the food delivery field, responses based on individual preferences, past behavioral history, and real-time emotional data are required to increase user satisfaction. The present invention aims to solve these problems.

[1504] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a device that collects personal data, a device that transmits the collected data to the server, a device that preprocesses the data received by the server and generates a user model using a machine learning algorithm, a device that generates a digital clone using the generated user model and transmits it to the terminal, a device that causes the digital clone to perform online tasks in accordance with the user's instructions, a device that recognizes emotional data in real time, and a device that analyzes the user's state using the recognized emotional data and suggests appropriate actions. This enables quick and accurate responses that take the user's emotional state into consideration, and in the food delivery field, it enables appropriate suggestions based on individual preferences and past behavioral history to increase user satisfaction.

[1505] "Personal data" refers to all data related to a user, such as the user's personal information, usage history, preference information, and allergy information.

[1506] A "server" is a computer system connected via a network that preprocesses data, runs machine learning algorithms, and generates user models and digital clones.

[1507] A "terminal" is a device used by a user, such as a computer or smartphone, that communicates with a server to receive a digital clone and realizes interaction with the user.

[1508] A "machine learning algorithm" is a computational method used to learn patterns and features in data and make predictions or classifications.

[1509] A "user model" is a digital profile that reflects a user's behavioral patterns and preferences, generated by machine learning algorithms.

[1510] A "digital clone" is a software agent that mimics a user's behavior and decisions, a virtual entity that performs online tasks according to the user's instructions.

[1511] "Emotion data" is data that indicates the emotional state of the user, obtained from the user's voice, facial expression, words, etc.

[1512] An "emotion engine" is a software module for recognizing and analyzing emotional data in real time.

[1513] "Online tasks" are all tasks that are performed using the Internet, and specifically include attending meetings, creating minutes, and managing schedules.

[1514] The "device for suggesting appropriate actions" is a device that recommends actions that are most suitable for the user's state based on collected emotional data and a user model.

[1515] "Food delivery" is a service that delivers food and drink to a location specified by the user.

[1516] The present invention relates to a system that uses a user's personal data and emotional data to generate a digital clone and assist with online tasks. The system collects personal data, transmits it to a server, performs data preprocessing, generates a user model using a machine learning algorithm, creates a digital clone, recognizes emotional data in real time, and suggests actions according to the user's state.

[1517] System configuration

[1518] The system includes the following components:

[1519] 1. Terminal devices for users to collect data

[1520] 2. A device where the server receives the data and generates a user model through preprocessing and machine learning algorithms.

[1521] 3. A device that uses a digital clone received by a terminal from a server to perform online tasks.

[1522] 4. A software module that enables the emotion engine to recognize and analyze emotion data in real time.

[1523] Explanation of program processing

[1524] The server receives personal and emotional data sent from the user's device. Data collection is performed using the smartphone's built-in microphone and camera, which capture voice and facial expression data. The received data is preprocessed on the server, and a user model is generated using a machine learning algorithm. This user model is a digital profile that reflects the user's past behavior and preferences. A digital clone is then generated based on this user model and sent to the user's device.

[1525] The emotion engine runs on the device and recognizes the user's real-time emotional state. The recognized emotional data is analyzed and suggestions appropriate to the user's state are made. For example, in a food delivery application, if a user is feeling stressed, the engine will suggest dishes that are suitable for relieving stress based on past preferences.

[1526] Usage example

[1527] Below are some specific examples of food delivery:

[1528] 1. The user types into the terminal, "I'm feeling stressed, so please give me some suggestions."

[1529] 2. The emotion engine detects the user's emotions in real time and recognizes "stress."

[1530] 3. The server analyzes the user's past order history and preferences based on the user model, and suggests dishes suitable for relieving stress.

[1531] 4. The device displays the suggested dishes to the user, and the user confirms the order.

[1532] 5. The server sends the order to the delivery service and manages the delivery status in real time.

[1533] Prompt Sentence Examples

[1534] An example of a prompt sentence for suggesting a dish to a user who is feeling stressed is as follows:

[1535] For example: "The user is currently feeling stressed. Their past ordering history shows that they like chocolate cake and comfort food to relieve stress. Please recommend appropriate dishes based on their current emotional state."

[1536] As described above, the system of the present invention can provide a fast and accurate response that takes into account the user's emotional state, which can increase user satisfaction, especially in the food delivery field. It can also provide high-level support using emotional data for other online tasks.

[1537] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1538] Step 1:

[1539] User collects data

[1540] Users use their smartphone's camera and microphone to collect personal and emotional data. Input data includes voice and facial expression data. This data is temporarily stored on the device. The collected voice and facial expression data is provided as output.

[1541] Step 2:

[1542] The device sends the data to the server

[1543] The device encrypts the collected personal data and emotion data and transmits them to the server via a secure protocol. The input data includes the collected voice and facial expression data. The output data includes the personal data and emotion data received by the server.

[1544] Step 3:

[1545] The server preprocesses the data

[1546] The server preprocesses the received data, removing noise and normalizing the data. The input data includes the server's personal data and emotion data sent from the device. After preprocessing, the cleaned data is stored on the server as the output.

[1547] Step 4:

[1548] The server generates the user model

[1549] The server uses the preprocessed data to generate a user model based on machine learning algorithms. The input data includes the preprocessed personal data and emotional data. The output is a user model, which is a digital profile that reflects the user's behavioral patterns and preferences.

[1550] Step 5:

[1551] The server creates a digital clone and sends it to the device.

[1552] The server creates a digital clone based on the generated user model and sends it to the terminal. The input data includes the user model. The output data includes the digital clone received by the terminal.

[1553] Step 6:

[1554] Emotion engine recognizes emotional data in real time

[1555] The emotion engine of the device recognizes the user's emotion data in real time. The input data includes real-time emotion data obtained from the user's voice and facial expressions. The output is the detected user's emotional state provided to the device.

[1556] Step 7:

[1557] The emotion engine analyzes the user's emotional state and makes appropriate suggestions to them.

[1558] The emotion engine on the device analyzes the recognized emotion data and the user model and makes suggestions appropriate to the user's state. For example, if a user is feeling stressed, it will suggest dishes that are good for relieving stress. The input data includes the detected emotional state and the user model. The output is a display of the suggestions made to the user.

[1559] Step 8:

[1560] The user confirms the order

[1561] The user confirms the order based on the suggestions displayed on the terminal. The input data includes the suggestions displayed on the terminal. The output is the confirmed order sent to the server.

[1562] Step 9:

[1563] The server processes the order and sends it to the delivery service

[1564] The server processes the user's confirmed order and sends it to the delivery service. The input data includes the user's confirmed order. The output data includes the order details received by the delivery service.

[1565] Step 10:

[1566] The server manages the delivery status

[1567] The server tracks the delivery status from the delivery service in real time and provides the information to the user. The input data includes delivery progress information from the delivery service. The output data includes delivery progress information provided to the user.

[1568] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1569] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1570] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1571] [Fourth embodiment]

[1572] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1573] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1574] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1575] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1576] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1577] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1578] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1579] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1580] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1581] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1582] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1583] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1584] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1585] The present invention relates to a system for creating a digital clone using personal data of a user and assisting with online tasks. Specific embodiments of the system are described below.

[1586] Digital Clone System Overview

[1587] The system of the present invention is composed of three main components: a "server," a "terminal," and a "user." Users interact with the system using dedicated applications. Data collection, processing, cloning, and task assistance are performed, and each step to achieve this is described in detail below.

[1588] User Data Collection

[1589] Users provide their own data, including email history, schedules, voice data, photos, internet history, etc. Users select this data through a dedicated application and upload it to the system.

[1590] The device temporarily stores this data and transmits it to a server in encrypted form using a secure protocol.

[1591] Data Preprocessing and Training

[1592] The server preprocesses the received data, which includes cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences and generate a user model. This model serves as the basis for creating a digital clone of the user.

[1593] Creating a digital clone

[1594] The server generates a digital clone based on the learned user model. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions.

[1595] The device stores the received digital clone locally and activates it when the user wants to use it.

[1596] Online task assistance

[1597] The user instructs the digital clone to perform online tasks, such as "attend tomorrow's online meeting and take minutes."

[1598] The device analyzes this instruction and requests the necessary information from the server, which then collects meeting information from the user's schedule management system and sends the details to the device.

[1599] The digital clone automatically logs in at the designated meeting time, performs real-time voice recognition, and converts the conversation into text. When the meeting ends, a summary of the meeting minutes is sent to the user via the device.

[1600] Specific examples

[1601] Example 1: Creating minutes for online meetings

[1602] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1603] 2. The device sends this instruction to the server.

[1604] 3. The server collects the meeting details (participant list, agenda, etc.) and sends them to the terminal.

[1605] 4. A digital clone logs in at the meeting time and recognizes and transcribes the conversation in real time.

[1606] 5. The digital clone compiles minutes and sends them to the user via the terminal after the meeting ends.

[1607] Example 2: Task Management

[1608] 1. The user asks, "Organize this week's tasks and set reminders."

[1609] 2. The device sends the user's schedule to the server.

[1610] 3. The server analyzes the user's schedule and identifies high-priority tasks.

[1611] 4. The server lists the identified tasks, sets reminders and sends them to the device.

[1612] 5. The device notifies the user of the reminder.

[1613] How to use

[1614] To use the system of the present invention, a user installs a dedicated application and registers as a user. Next, the user selects and provides the data they want to collect, and begins creating a digital clone. Once the digital clone is created, the user can instruct the digital clone to perform various online tasks through the dedicated application, improving work efficiency.

[1615] In this way, by implementing the digital clone system of the present invention, users can efficiently manage and carry out their own work, thereby reducing their workload.

[1616] The processing flow will be explained below.

[1617] Digital clone system processing flow

[1618] Data Collection Phase

[1619] Step 1:

[1620] The user launches the dedicated application and selects the data they want to collect (emails, schedules, audio data, photos, internet history, etc.).

[1621] Step 2:

[1622] The device temporarily stores selected data, for example, email history in a temporary folder in local storage.

[1623] Step 3:

[1624] The device encrypts the stored data using a secure communication protocol (e.g., SSL / TLS) and sends it to the server.

[1625] Data preprocessing and training phase

[1626] Step 4:

[1627] The server preprocesses the data it receives. Specifically, it cleans and filters the data to remove unnecessary noise. For example, it removes spam and advertising emails from email data.

[1628] Step 5:

[1629] The server feeds the preprocessed data into machine learning algorithms to generate a user model, a process that learns and models the user's behavioral patterns and preferences.

[1630] Digital Clone Generation Phase

[1631] Step 6:

[1632] The server generates a digital clone based on the learned user model. This digital clone is an agent that simulates the user's actions and decisions.

[1633] Step 7:

[1634] The server converts the generated digital clone into binary format and transmits it to the terminal.

[1635] Step 8:

[1636] The device stores the received digital clone locally and launches it when the user needs it.

[1637] Online task assistance phase

[1638] Step 9:

[1639] The user gives the digital clone tasks through a dedicated application, such as "attend tomorrow's online meeting and take minutes."

[1640] Step 10:

[1641] The terminal analyzes the user's instructions and sends a request to the server to obtain the required information.

[1642] Step 11:

[1643] The server collects the user's schedule and meeting information and sends it to the device, for example extracting meeting details from the user's calendar.

[1644] Step 12:

[1645] Based on the information acquired by the device, a digital clone is activated and allowed to participate in the meeting. The digital clone converts speech into text in real time and creates minutes.

[1646] Step 13:

[1647] After the meeting, the digital clone compiles minutes and sends them to the user via the device.

[1648] Step 14:

[1649] The user reviews the minutes they receive and gives the digital clone further instructions, such as "Share the minutes with the team."

[1650] Specific examples

[1651] Example 1: Process flow for creating minutes of an online meeting

[1652] Step 1:

[1653] The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1654] Step 2:

[1655] The terminal analyzes the instructions and sends them to the server.

[1656] Step 3:

[1657] The server collects the details of the meeting (participant list, agenda, etc.) and sends them to the terminal.

[1658] Step 4:

[1659] The digital clone automatically logs in at meeting time and recognizes voice and transcribes the conversation in real time.

[1660] Step 5:

[1661] The digital clone compiles minutes and sends them to the user via the terminal after the meeting ends.

[1662] Example 2: Task management process flow

[1663] Step 1:

[1664] The user says, "Organize this week's tasks and set reminders."

[1665] Step 2:

[1666] The terminal transmits the user's schedule information to the server.

[1667] Step 3:

[1668] The server analyzes the schedule and identifies important and urgent tasks.

[1669] Step 4:

[1670] The server lists the identified tasks and sends them to the terminal.

[1671] Step 5:

[1672] The device notifies the user of the reminder.

[1673] These detailed processing steps allow the digital clone system of the present invention to efficiently perform tasks according to the user's instructions, thereby reducing the burden on the user.

[1674] Example 1

[1675] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1676] In today's world, with the increase in online tasks, it is becoming increasingly difficult for individual users to efficiently manage their time and complete tasks. Furthermore, taking minutes of online meetings and managing tasks is time-consuming and accuracy is an issue. This has led users to seek ways to improve the efficiency of their work.

[1677] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1678] In this invention, the server includes means for collecting personal data, means for transmitting the collected data to the server using an encryption protocol, means for the server to preprocess the received data and generate a user model using a machine learning algorithm, means for the server to create a digital clone using the generated user model and transmit it to the terminal, means for the digital clone to analyze the user's instructions using natural language processing and execute online tasks, and means for the digital clone to notify the user of the results after the task is executed. This enables users to efficiently manage and execute online tasks, significantly improving work efficiency.

[1679] "Personal data" refers to any form of information related to a user (email history, schedule, audio data, photos, internet history, etc.).

[1680] "Cryptographic protocol" refers to the technology (e.g., TLS) used to ensure the secure communication of data.

[1681] A "server" refers to a computer system that performs core computations, such as data preprocessing, generating machine learning models, and creating digital clones.

[1682] A "machine learning algorithm" refers to a computational method that learns patterns from data and generates and updates a user model.

[1683] A "user model" refers to a data structure that represents a user's behavioral patterns and preferences, generated by a machine learning algorithm based on collected personal data.

[1684] A "digital clone" refers to a software agent that is generated based on a user model and mimics the user's behavior and decisions.

[1685] "Terminal" refers to the device (e.g., smartphone, tablet, PC) through which a user interacts with their digital clone.

[1686] "Natural language processing" refers to the technology that a digital clone uses to understand and interpret user instructions.

[1687] "Online tasks" refer to Internet tasks that a user instructs a digital clone to perform (e.g., participating in an online meeting, task management).

[1688] "Speech recognition technology" refers to technology that converts speech into text in real time.

[1689] "Minutes" refers to a document that records the contents of the conversation that took place during an online meeting as text.

[1690] "Reminder" refers to a function that notifies the user according to a schedule.

[1691] "Information collection system" refers to external APIs and tools for collecting user schedules and other necessary information.

[1692] The present invention relates to a system for creating a digital clone using personal data of a user and assisting with online tasks. Specific embodiments of the system are described below.

[1693] User Data Collection

[1694] Users use a dedicated application to provide their own data, including email history, schedules, voice data, photos, internet history, etc. The user selects this data and uploads it to the system.

[1695] The device temporarily stores this data and transmits it to the server using an encryption protocol such as TLS (Transport Layer Security).

[1696] Data Preprocessing and Training

[1697] The server preprocesses the received data, including cleaning the data (e.g., imputing missing values ​​and removing outliers), filtering, and normalization. This preprocessing is performed using Python's Pandas and Numpy libraries.

[1698] The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences, generating a user model using libraries like TensorFlow and PyTorch. This user model serves as the basis for creating a digital clone of the user.

[1699] Creating a digital clone

[1700] The server generates a digital clone based on the learned user model. This digital clone is a software agent generated using Python scripts that mimics the user's behavior and decisions. The generated digital clone is converted into a binary format and sent to the device.

[1701] The terminal stores the received digital clone in local storage, using SQLite or JSON format. When the user wants to use it, the terminal launches the digital clone, possibly using a Python script or a Docker container.

[1702] Online task assistance

[1703] The user instructs the digital clone to perform online tasks. These instructions are entered as prompt sentences through the interface of a dedicated application. For example, the user might instruct the clone to "attend tomorrow's online meeting and take minutes."

[1704] The device parses this instruction using natural language processing (NLP) techniques (e.g., spaCy or NLTK libraries). The device then requests the necessary information from the server. For example, it could retrieve meeting information from the Google Calendar API.

[1705] The server collects meeting information from the user's scheduler and sends the details to the device. The digital clone then automatically logs in at the designated meeting time and performs real-time speech recognition and converts the conversation into text, using, for example, the Google Cloud Speech-to-Text API. Once the meeting is over, a summary of the meeting minutes is reported to the user via the device.

[1706] Specific examples

[1707] Example 1: Creating minutes for online meetings

[1708] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1709] 2. The device analyzes this instruction and requests the necessary information from the server.

[1710] 3. The server uses the Google Calendar API to collect the meeting details and send them to the device.

[1711] 4. At the meeting time, the digital clone logs in using Selenium and converts the audio to text using the Google Cloud Speech-to-Text API.

[1712] 5. The digital clone compiles the minutes in Markdown format and notifies the user through a dedicated application.

[1713] Example 2: Task Management

[1714] 1. The user asks, "Organize this week's tasks and set reminders."

[1715] 2. The terminal sends the user's schedule information to the server.

[1716] 3. The server uses Natural Language Processing (NLP) technology to analyze the schedule and identify high-priority tasks.

[1717] 4. The server lists the identified tasks, sets reminders using the Google Calendar API, and sends them to the device.

[1718] 5. The device notifies the user of the reminder.

[1719] Prompt Sentence Examples

[1720] "Please take the minutes of the meeting tomorrow at 9 o'clock."

[1721] "Organize my tasks for this week and set reminders."

[1722] By following this specific process flow, users can efficiently manage and complete online tasks.

[1723] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1724] Step 1:

[1725] The user launches a dedicated application and provides their personal data (e.g., email history, schedule, audio data, photos, internet history).

[1726] Input: Personal Data

[1727] Output: Temporarily saved dataset

[1728] Specific operations: Select the type of data to provide through the dedicated application interface and press the button to upload the data.

[1729] Step 2:

[1730] The device temporarily stores the data uploaded by the user and sends it to the server using an encryption protocol such as TLS (Transport Layer Security).

[1731] Input: Temporarily saved dataset

[1732] Output: Data in encrypted form

[1733] Specific operation: The data is temporarily stored in a local database (e.g., SQLite) on the device, and then encrypted using the OpenSSL library. After encryption, the data is sent to the server.

[1734] Step 3:

[1735] The server stores the received data in a cloud database (e.g., Amazon RDS) and performs preprocessing.

[1736] Input: Encrypted data

[1737] Output: Preprocessed dataset

[1738] Specific operations: Decrypt the data and use the Pandas library to clean (fill in missing values, remove outliers), filter, and normalize the data.

[1739] Step 4:

[1740] The server inputs the preprocessed data into a machine learning algorithm (e.g., TensorFlow, PyTorch) to generate a user model.

[1741] Input: Preprocessed dataset

[1742] Output: User model

[1743] Specific operation: Data is fed into a machine learning algorithm to learn behavioral patterns and preferences. As a result, a user model is generated.

[1744] Step 5:

[1745] The server generates a digital clone based on the user model, converts it into binary format, and transmits it to the terminal.

[1746] Input: User model

[1747] Output: Digital clone in binary format

[1748] Specific operation: The user model is embodied as a digital clone using a Python script, then converted into binary format. The converted digital clone is sent to the terminal.

[1749] Step 6:

[1750] The terminal stores the received digital clone in local storage (e.g., SQLite, JSON format) and activates the digital clone as needed.

[1751] Input: Digital clone in binary format

[1752] Output: Activated digital clone

[1753] Specific operation: A digital clone is stored in local storage and launched using a Python script or Docker container when the user uses it.

[1754] Step 7:

[1755] The user instructs the digital clone on tasks through a dedicated application.

[1756] Input: Prompt sentence (e.g., "Please attend tomorrow's online meeting and take minutes.")

[1757] Output: Parsed task instructions

[1758] Specific operations: Enter the prompt text through the dedicated application interface and press the send button.

[1759] Step 8:

[1760] The device analyzes the user's instructions using natural language processing (NLP) technology (e.g., spaCy, NLTK) and requests the necessary information from the server.

[1761] Input: prompt statement

[1762] Output: Task execution information request

[1763] Specific operations: Analyzes the text of the user's instructions using NLP technology, interprets the task content, and sends a request to the server for the appropriate information.

[1764] Step 9:

[1765] The server collects the necessary information from the user's schedule management system (e.g., Google Calendar API) and sends it to the device.

[1766] Input: Task execution information request

[1767] Output: Collected schedule information

[1768] Specific operation: Uses API to obtain user schedule information and sends it to the device.

[1769] Step 10:

[1770] The digital clone performs online tasks (e.g., participating in an online meeting) and notifies the user of the results.

[1771] Input: Collected schedule information

[1772] Output: Task results (e.g. meeting minutes)

[1773] Specific operations: Log in at the specified meeting time (e.g., using Selenium), convert the audio into text using the Google Cloud Speech-to-Text API, summarize the task results, and notify the user.

[1774] Prompt Sentence Examples

[1775] "Please take the minutes of the meeting tomorrow at 9 o'clock."

[1776] "Organize my tasks for this week and set reminders."

[1777] The above is the specific processing flow of this system.

[1778] (Application example 1)

[1779] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1780] While existing technologies exist that generate digital clones based on a user's personal data and assist with online tasks, they lack the means to analyze a customer's purchase history and preferences in physical stores and recommend appropriate products. As a result, improvements to customer service and the shopping experience in physical stores have not been fully realized. The present invention aims to solve this problem and improve the customer experience in physical stores.

[1781] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1782] In this invention, the server includes a means for acquiring personal data, a means for transmitting the acquired data to the server, a means for the server to preprocess the received data and generate a user model using a machine learning algorithm, a means for creating a digital clone using the generated user model and transmitting it to the terminal, a means for the digital clone to perform online tasks according to the user's instructions, and a means for the digital clone to analyze the customer's purchase history and preferences using smart glasses in a physical store and recommend appropriate products. This enables the digital clone to immediately respond to customer needs even in a physical store and recommend optimal products.

[1783] "Personal data" refers to data that includes a user's behavioral history, purchasing history, preferences, and other personal information.

[1784] A "server" is a computer system that receives user data, pre-processes it, generates models using machine learning, and creates and transmits digital clones.

[1785] A "terminal" is a device that receives a digital clone sent from a server and allows a user to interact with the digital clone.

[1786] A "machine learning algorithm" is a technology that analyzes user data and learns user behavior patterns and preferences.

[1787] A "user model" is a data model that reflects a user's behavioral patterns and preferences, generated using a machine learning algorithm.

[1788] A "digital clone" is a software agent that mimics a user's behavior and decisions based on a generated user model.

[1789] An "online task" is a task performed over the Internet and executed by the digital clone based on the user's instructions.

[1790] A "physical store" is a physically existing sales store that is a place that users (customers) visit in person.

[1791] "Smart glasses" are wearable devices that have augmented reality and information display functions and are used by store clerks in physical stores when providing customer service.

[1792] "Purchase history" is a record of products that a customer has purchased in the past.

[1793] "Preferences" refer to a user's tendency to prefer a particular type or style of product.

[1794] In the context of "Levidence," a "designated document" is a document submitted in the course of a trial or investigation that is referenced based on specific requirements.

[1795] The present invention is a system for collecting personal data of users and generating digital clones, and it is particularly intended to provide personalized product recommendations to customers using smart glasses in brick-and-mortar stores. Specific embodiments of the system are described below.

[1796] System Overview

[1797] The system of the present invention consists of a server, a terminal, and a user. The user interacts with the system using smart glasses, and data collection, processing, digital cloning, and product recommendations are performed. This procedure is described in detail below.

[1798] Retrieving User Data

[1799] Users provide their own data, including purchase history, preferences, and other behavioral patterns. They select this data through a dedicated application and upload it to the system. The device temporarily stores this data and transmits it to the server in encrypted form using a secure protocol.

[1800] Data preprocessing and user model generation

[1801] The server preprocesses the received data, which includes cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms (e.g., the scikit-learn library) to learn the user's behavioral patterns and preferences and generate a user model.

[1802] Creating and transmitting digital clones

[1803] The server generates a digital clone based on the learned user model. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions. The device stores the received digital clone locally and activates it when the user uses it.

[1804] Product recommendations in physical stores

[1805] A user wears smart glasses and performs the task of recommending products to a customer. For example, a customer may request, "Please show me the latest new products." The device sends this instruction to the server, which analyzes the customer's purchase history and preferences. Based on the analysis results, the device sends a recommendation list of appropriate new products. The user can then suggest the best products to the customer based on the information displayed on the smart glasses' display.

[1806] Specific examples

[1807] Usage Scenarios

[1808] 1. Customer A asks the store clerk to show him the latest new products.

[1809] 2. The device sends this instruction to the server.

[1810] 3. The server analyzes customer A's purchase history and preferences, generates a list of recommended products, and sends it to the terminal.

[1811] 4. The user (store clerk) introduces the recommended product to customer A through the smart glasses.

[1812] Prompt Sentence Examples

[1813] "Customer ID: customer123. Looking at their recent purchase history, they have purchased a bag, shoes, and a jacket. New items have arrived. What products would you recommend for this customer?"

[1814] Hardware and software used

[1815] Hardware: Servers (e.g., Amazon Web Services, Google Cloud Platform)

[1816] Software: Python libraries (e.g., requests, scikit-learn)

[1817] This will enable the digital clone to instantly respond to customer needs in physical stores and recommend the most suitable products.

[1818] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1819] Step 1:

[1820] User provides data

[1821] Users select data such as their purchase history, preferences, and behavioral patterns through a dedicated application and upload that data to the device. INPUT is the user's personal information and history data, and OUTPUT is encrypted data stored on the device.

[1822] Step 2:

[1823] The device sends data to the server

[1824] The device temporarily stores the acquired data and sends it to the server in encrypted form using a secure protocol (e.g., HTTPS). INPUT is the encrypted data stored on the device, and OUTPUT is the encrypted data sent to the server.

[1825] Step 3:

[1826] The server preprocesses the data

[1827] The server performs pre-processing on the received data, such as cleaning, filtering, and normalization. Unwanted noise is removed and the data is in a format that can be analyzed. INPUT is the encrypted data sent to the server, and OUTPUT is the pre-processed data.

[1828] Step 4:

[1829] The server generates a user model using a machine learning algorithm

[1830] The server uses the preprocessed data to apply machine learning algorithms (e.g., scikit-learn) to learn the user's behavioral patterns and preferences, which results in a user model. The input is the preprocessed data, and the output is the user model.

[1831] Step 5:

[1832] The server uses the user model to create a digital clone

[1833] The server creates a digital clone based on the generated user model. The created digital clone is converted into binary format and sent to the terminal. The input is the user model, and the output is the binary format digital clone.

[1834] Step 6:

[1835] The device stores the digital clone locally and activates it when the user wants to use it.

[1836] The terminal stores the received digital clone locally and activates it as needed. This allows the user to receive services from the digital clone. INPUT is the digital clone sent from the server, and OUTPUT is the digital clone stored in the terminal.

[1837] Step 7:

[1838] Users wear smart glasses to serve customers

[1839] The user (store clerk) wears smart glasses and receives requests from customers (e.g., "Please show me the latest new products"). INPUT is the customer's request, and OUTPUT is the instruction sent to the terminal.

[1840] Step 8:

[1841] The terminal sends the customer's request to the server

[1842] The terminal analyzes the customer's request and sends the information to the server. INPUT is the customer's request, and OUTPUT is the request to the server.

[1843] Step 9:

[1844] The server analyzes the customer's purchase history and preferences to recommend appropriate products.

[1845] The server analyzes the customer's purchase history and preferences based on the received request. This generates an appropriate product recommendation list and sends it to the terminal. INPUT is the request to the server and customer data, and OUTPUT is the list of recommended products.

[1846] Step 10:

[1847] Users can suggest products to customers through smart glasses

[1848] The user (store clerk) recommends the best products to the customer based on the product recommendation list displayed on the smart glasses display. The input is the list of recommended products, and the output is the product proposal to the customer.

[1849] This allows the digital clone to instantly respond to customer needs in a physical store and recommend the most suitable products.

[1850] "Customer ID: customer123. Looking at their recent purchase history, they have purchased a bag, shoes, and a jacket. New items have arrived. What products would you recommend for this customer?"

[1851] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1852] The present invention relates to a system for creating a digital clone using personal data and emotional data of a user and assisting the user in online tasks. Specific embodiments of the system are described below.

[1853] Digital Clone System Overview

[1854] The system of the present invention consists of four main components: a "server," a "terminal," a "user," and an "emotion engine." The user interacts with the system using a dedicated application, which collects data, processes it, creates clones, and assists with tasks. The emotion engine also recognizes the user's emotions in real time and utilizes this information throughout the system.

[1855] User and sentiment data collection

[1856] The user selects the data they want to collect (email, schedule, voice data, photos, internet history, etc.) through a dedicated application, and simultaneously collects emotional data (voice and facial expression data). Emotional data is acquired through sensors such as a microphone and camera.

[1857] The device temporarily stores this data and transmits it to a server in encrypted form using a secure protocol.

[1858] Data Preprocessing and Training

[1859] The server preprocesses the received data. This involves cleaning, filtering, and normalizing the data to remove unwanted noise. The server then uses machine learning algorithms to learn the user's behavioral patterns and preferences and generate a user model. This model serves as the basis for creating a digital clone of the user.

[1860] The server similarly processes the emotion data obtained through the emotion engine and incorporates the user's emotional state into the model.

[1861] Creating a digital clone

[1862] The server generates a digital clone based on the trained user model and emotional data. The generated digital clone is converted into binary format and sent to the device. This digital clone is a software agent that mimics the user's behavior and decisions, and responds taking into account the user's emotional state.

[1863] The device stores the received digital clone locally and activates it when the user wants to use it.

[1864] Online task assistance

[1865] The user instructs the digital clone to perform online tasks, such as "attend tomorrow's online meeting and take minutes."

[1866] The device analyzes this instruction and requests the necessary information from the server, which then collects meeting information from the user's schedule management system and sends the details to the device.

[1867] The digital clone automatically logs in at the designated meeting time, performs real-time speech recognition, and converts the conversation into text. When the meeting ends, a summary of the meeting minutes is sent to the user via their device. In addition, an emotion engine monitors the user's emotional state during the meeting and creates appropriate responses and meeting minutes based on that information.

[1868] Specific examples

[1869] Example 1: Creating minutes for online meetings

[1870] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1871] 2. The device sends this instruction to the server.

[1872] 3. The server collects the meeting details (participant list, agenda, etc.) and sends them to the terminal.

[1873] 4. The digital clone automatically logs in at the meeting time, recognizes the conversation in real time, and transcribes it into text. The emotion engine simultaneously monitors the person's emotional state.

[1874] 5. The digital clone compiles minutes of the meeting, taking into account emotional data, and sends them to the user via their device after the meeting ends.

[1875] Example 2: Task Management

[1876] 1. The user asks, "Organize this week's tasks and set reminders."

[1877] 2. The terminal sends the user's schedule information to the server.

[1878] 3. The server analyzes the schedule and identifies important and urgent tasks, taking into account emotional data.

[1879] 4. The server lists the identified tasks and sends them to the device.

[1880] 5. The device notifies the user of the reminder. The emotion engine provides feedback on the user's emotional state and adjusts the wording of the reminder.

[1881] How to use

[1882] To use the system of the present invention, a user installs a dedicated application and registers as a user. Next, the user selects and provides the data they want to collect, and begins creating a digital clone. Once the digital clone is created, the user can instruct the digital clone to perform various online tasks through the dedicated application, improving work efficiency.

[1883] Furthermore, the introduction of an emotion engine enables the digital clone to respond and act in a way that takes into account the user's emotional state, providing more appropriate and effective support.

[1884] In this way, by implementing the digital clone system of the present invention, users can efficiently manage and carry out their own work, reducing their workload. The introduction of an emotion engine will provide even more advanced support.

[1885] The processing flow will be explained below.

[1886] Digital clone system processing flow

[1887] Data Collection Phase

[1888] Step 1:

[1889] The user launches the dedicated application and selects the data they want to collect (emails, schedules, audio data, photos, internet history, etc.).

[1890] Step 2:

[1891] The device temporarily stores the selected data, while simultaneously collecting the user's voice and facial expression data using the microphone and camera.

[1892] Step 3:

[1893] The device analyzes the emotional state using voice and facial expression data and generates emotional data.

[1894] Step 4:

[1895] The personal data and emotional data collected by the device are sent to a server using a secure protocol (e.g., encryption using SSL / TLS).

[1896] Data preprocessing and training phase

[1897] Step 5:

[1898] The server preprocesses the received personal and emotional data, specifically cleaning, filtering, and denoising the data.

[1899] Step 6:

[1900] The server then feeds the pre-processed data into machine learning algorithms to generate a user model that reflects the user's behavioral patterns and emotional state.

[1901] Digital Clone Generation Phase

[1902] Step 7:

[1903] The server generates a digital clone based on the learned user model and emotional data. The digital clone simulates the user's behavior and decisions and responds according to the user's emotions.

[1904] Step 8:

[1905] The server converts the generated digital clone into binary format and transmits it to the terminal.

[1906] Step 9:

[1907] The device stores the received digital clone locally and activates it when the user gives the command.

[1908] Online task assistance phase

[1909] Step 10:

[1910] The user issues tasks to the digital clone through a dedicated application, such as "attend tomorrow's online meeting and take minutes."

[1911] Step 11:

[1912] The terminal analyzes the user's instructions and sends a request to the server to obtain the required information.

[1913] Step 12:

[1914] The server collects the user's schedule and meeting information and sends it to the terminal.

[1915] Step 13:

[1916] The device activates a digital clone based on the information it acquires and automatically logs in at the specified meeting time.

[1917] Step 14:

[1918] The digital clone converts speech to text in real time during meetings and creates meeting minutes, while the emotion engine monitors the user's emotional state and suggests appropriate responses.

[1919] Step 15:

[1920] After the meeting, the digital clone will compile minutes and report them to the user via their device, taking into account emotional data.

[1921] Specific examples

[1922] Example 1: Process flow for creating minutes of an online meeting

[1923] Step 1:

[1924] The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1925] Step 2:

[1926] The terminal sends the instruction to the server.

[1927] Step 3:

[1928] The server collects the details of the meeting (participant list, agenda, etc.) and sends them to the terminal.

[1929] Step 4:

[1930] A digital clone logs in at meeting time, converts speech to text in real time, and creates minutes. An emotion engine monitors the user's emotional state during the meeting.

[1931] Step 5:

[1932] The digital clone will compile minutes and report them to the user via their device after the meeting, taking into account emotional data.

[1933] Example 2: Task management process flow

[1934] Step 1:

[1935] The user says, "Organize this week's tasks and set reminders."

[1936] Step 2:

[1937] The terminal transmits the user's schedule information and emotion data to the server.

[1938] Step 3:

[1939] The server analyzes schedule and emotion data to identify important and urgent tasks.

[1940] Step 4:

[1941] The server lists the identified tasks, generates reminders based on the user's emotional state, and sends them to the device.

[1942] Step 5:

[1943] The device notifies the user of the reminder, and the emotion engine monitors the user's emotional state and adjusts the wording of the reminder accordingly.

[1944] These detailed processing steps enable the digital clone system of the present invention to efficiently perform tasks according to the user's instructions and provide appropriate support that takes into account the user's emotional state.

[1945] Example 2

[1946] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1947] While conventional digital assistant systems could assist users with online tasks using their personal data, they lacked the ability to adjust responses and behaviors based on the user's emotional state. This resulted in problems such as stress and insufficient support. Furthermore, they lacked the ability to respond in real time for specific online tasks, such as taking meeting minutes. Furthermore, prioritizing tasks and setting reminders did not take the user's emotional state into account, which meant they did not fully meet the user's needs.

[1948] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for collecting personal data and emotional data; means for encrypting the collected data and transmitting it to the server; means for the server to preprocess the received data and generate a user model using a machine learning algorithm; means for creating a digital clone based on the generated user model and emotional data and transmitting it to the terminal; and means for the digital clone to perform online tasks according to the user's instructions and adjust responses and report content using an emotion engine. This makes it possible to provide more personalized assistance by utilizing the user's personal data and emotional data, and to realize responses and actions appropriate to the user's emotional state. Furthermore, appropriate services that take the user's emotions into consideration can be provided when efficiently completing specific online tasks and prioritizing tasks.

[1949] "Personal data" refers to all information about the user, including emails, schedules, audio data, photos, internet history, and so on.

[1950] "Emotion data" refers to data that reflects the user's emotional state, and includes information such as voice and facial expressions acquired using a microphone or camera.

[1951] "Server" refers to the computer system that collects, preprocesses, analyzes, generates models, and creates digital clones of data.

[1952] "Terminal" refers to a device used by a user to operate the system, including smartphones and personal computers.

[1953] "Encryption" refers to the process of converting data into a special format for secure communication.

[1954] "Preprocessing" refers to processes such as data cleaning, filtering, and noise removal that transform raw data into a format suitable for machine learning algorithms.

[1955] A "machine learning algorithm" refers to a computational method that learns patterns from data and uses them to make predictions and classifications.

[1956] "User model" refers to a digital representation of a user's behavior and preferences that is generated based on collected personal and emotional data.

[1957] A "digital clone" refers to a software agent that is created based on a user model and is used to carry out user instructions and tasks.

[1958] "Emotion engine" refers to a function that analyzes user emotional data and utilizes that information throughout the system.

[1959] "Online tasks" refer to work or tasks carried out over the Internet, including online meetings, taking minutes, and schedule management.

[1960] The present invention relates to a system that uses a user's personal data and emotional data to create a digital clone and assist with online tasks. This system mainly consists of four components: a "server," a "terminal," a "user," and an "emotion engine." The following describes the specific means and process for implementing the present invention.

[1961] 1. Initial Setup and Data Collection

[1962] Users install the dedicated application and set up their account. Next, they select the type of data to be collected (email, schedule, voice data, photos, internet history, etc.) and set the corresponding access permissions for the application. Emotional data is collected using the microphone and camera.

[1963] For example, a user launches a dedicated smartphone app and registers the day's calendar information and schedule. At the same time, the camera captures their facial expressions and the microphone records their mood for the day.

[1964] 2. Data Preprocessing and Encryption

[1965] The device temporarily stores the collected data and performs preprocessing using the Python Pandas library to clean and denoise the data. After preprocessing, the data is encrypted using AES encryption and sent to the server using a secure protocol (e.g., HTTPS).

[1966] 3. Training and Model Generation

[1967] The server analyzes the received data and generates a user model using machine learning algorithms (e.g., scikit-learn or TensorFlow). Python libraries such as Pandas and NumPy are used for data cleaning and normalization. Emotion data is also processed through an emotion engine, which incorporates the user's emotional state into the model.

[1968] 4. Creating a Digital Clone

[1969] The server generates a digital clone based on the trained user model and emotion data. This digital clone is converted into binary format and sent to the device. The device stores the received digital clone locally and activates it when the user uses it.

[1970] 5. Assistance and execution of tasks

[1971] The user gives the digital clone instructions, such as "Take minutes of tomorrow's online meeting." The device analyzes these instructions, sends them to the server, and retrieves the necessary information. The digital clone then participates in the online meeting, converts speech to text in real time, and generates minutes. It uses an emotion engine to adjust responses and reports based on the user's emotional state.

[1972] Specific examples

[1973] Example 1: Creating minutes for online meetings

[1974] 1. The user instructs the digital clone to "take minutes of the meeting tomorrow at 9 o'clock."

[1975] 2. The device sends this instruction to the server.

[1976] 3. The server collects the meeting details (participant list, agenda) and sends them to the terminal.

[1977] 4. The digital clone automatically logs in at the meeting time and converts the conversation into text using voice recognition. The emotion engine monitors the user's emotional state during the meeting and finalizes the minutes.

[1978] 5. The digital clone compiles the minutes and sends them to the user via the terminal.

[1979] Example 2: Task Management

[1980] 1. The user asks, "Organize this week's tasks and set reminders."

[1981] 2. The terminal sends the user's schedule information to the server.

[1982] 3. The server analyzes the schedule data and identifies important and urgent tasks, taking into account emotional data.

[1983] 4. The server lists the identified tasks and sends them to the device.

[1984] 5. The device notifies the user of the reminder. The emotion engine provides feedback on the user's emotional state and adjusts the wording of the reminder.

[1985] Prompt Sentence Examples

[1986] "Describe a system where a user provides data using a dedicated app to generate a digital clone along with emotional data."

[1987] In this way, the digital clone system of the present invention can significantly improve users' work efficiency by analyzing users' personal data and emotional data in real time and providing optimal online task support. The introduction of an emotion engine enables responses and actions that take the user's emotional state into account, providing more effective and personalized support.

[1988] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1989] Step 1:

[1990] Users install the dedicated application and set up an account. After launching the application, they select the type of data to be collected (email, schedule, voice data, photos, internet history, etc.) and set the corresponding access permissions for the application. Emotional data is also collected using the microphone and camera.

[1991] Input: User's personal and emotional data

[1992] Output: Temporarily save collected data to the device

[1993] For example, a user launches a dedicated smartphone app and allows access to calendar information and the microphone. The app then captures facial expressions with the camera and records the user's mood for the day with the microphone.

[1994] Step 2:

[1995] The terminal temporarily stores the collected data. Then, the data is cleaned to remove unnecessary noise and invalid entries. Specifically, the data is cleaned using the Python Pandas library. The data is also AES encrypted and sent to the server using a secure protocol (e.g., HTTPS).

[1996] Input: User's personal and emotional data

[1997] Output: Preprocessed encrypted data sent to the server

[1998] As a specific example, the terminal receives "schedule data" and "audio data," removes noise, applies AES encryption, and sends it to the server via HTTPS.

[1999] Step 3:

[2000] The server analyzes the received data and further cleans and normalizes it. Specifically, it processes the data using Pandas and NumPy. It then generates a user model using machine learning algorithms (e.g., scikit-learn and TensorFlow). It also analyzes the emotional data and incorporates the user's emotional state into the model via an emotion engine.

[2001] Input: Encrypted user personal and emotional data

[2002] Output: Integrated data including user model and emotional state

[2003] For example, the server normalizes large amounts of schedule data and voice data, analyzes them with machine learning algorithms, and generates a user model. Emotional data is also analyzed and reflected in the user model.

[2004] Step 4:

[2005] The server generates a digital clone based on the trained user model and emotion data. This digital clone is converted into binary format and sent to the device. The device stores the received digital clone locally and activates it when the user uses it.

[2006] Input: User model and emotion data

[2007] Output: Sending the binary digital clone to the device

[2008] As a concrete example, the server generates a digital clone based on the learned user model, encodes it as binary code, and sends it to the device, which then stores the received digital clone in local storage.

[2009] Step 5:

[2010] The user instructs the digital clone to perform an online task. For example, they give specific instructions such as, "Attend tomorrow's online meeting and take minutes." The device analyzes this instruction and sends it to the server. The server collects the necessary information and sends it back to the device. The digital clone then carries out the task and adjusts its responses and reports using an emotion engine.

[2011] Input: User instructions

[2012] Output: Task execution results and report to the user

[2013] As a concrete example, a user might say to the digital clone, "Take minutes of the meeting tomorrow at 9 o'clock." The device analyzes the instruction and sends it to the server. The server collects calendar information, and the digital clone automatically logs in to the meeting and creates minutes in real time using voice recognition. The emotion engine monitors emotions during the meeting and compiles appropriate minutes.

[2014] (Application example 2)

[2015] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2016] Conventional online task assistance systems do not take into account the user's emotional state, making it difficult to provide appropriate responses tailored to the user's actual needs and situation. Furthermore, they are unable to effectively utilize the user's preferences and past behavioral history, limiting the amount of suggestions and support they can provide. In particular, in the food delivery field, responses based on individual preferences, past behavioral history, and real-time emotional data are required to increase user satisfaction. The present invention aims to solve these problems.

[2017] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a device that collects personal data, a device that transmits the collected data to the server, a device that preprocesses the data received by the server and generates a user model using a machine learning algorithm, a device that generates a digital clone using the generated user model and transmits it to the terminal, a device that causes the digital clone to perform online tasks in accordance with the user's instructions, a device that recognizes emotional data in real time, and a device that analyzes the user's state using the recognized emotional data and suggests appropriate actions. This enables quick and accurate responses that take the user's emotional state into consideration, and in the food delivery field, it enables appropriate suggestions based on individual preferences and past behavioral history to increase user satisfaction.

[2018] "Personal data" refers to all data related to a user, such as the user's personal information, usage history, preference information, and allergy information.

[2019] A "server" is a computer system connected via a network that preprocesses data, runs machine learning algorithms, and generates user models and digital clones.

[2020] A "terminal" is a device used by a user, such as a computer or smartphone, that communicates with a server to receive a digital clone and realizes interaction with the user.

[2021] A "machine learning algorithm" is a computational method used to learn patterns and features in data and make predictions or classifications.

[2022] A "user model" is a digital profile that reflects a user's behavioral patterns and preferences, generated by machine learning algorithms.

[2023] A "digital clone" is a software agent that mimics a user's behavior and decisions, a virtual entity that performs online tasks according to the user's instructions.

[2024] "Emotion data" is data that indicates the emotional state of the user, obtained from the user's voice, facial expression, words, etc.

[2025] An "emotion engine" is a software module for recognizing and analyzing emotional data in real time.

[2026] "Online tasks" are all tasks that are performed using the Internet, and specifically include attending meetings, creating minutes, and managing schedules.

[2027] The "device for suggesting appropriate actions" is a device that recommends actions that are most suitable for the user's state based on collected emotional data and a user model.

[2028] "Food delivery" is a service that delivers food and drink to a location specified by the user.

[2029] The present invention relates to a system that uses a user's personal data and emotional data to generate a digital clone and assist with online tasks. The system collects personal data, transmits it to a server, performs data preprocessing, generates a user model using a machine learning algorithm, creates a digital clone, recognizes emotional data in real time, and suggests actions according to the user's state.

[2030] System configuration

[2031] The system includes the following components:

[2032] 1. Terminal devices for users to collect data

[2033] 2. A device where the server receives the data and generates a user model through preprocessing and machine learning algorithms.

[2034] 3. A device that uses a digital clone received by a terminal from a server to perform online tasks.

[2035] 4. A software module that enables the emotion engine to recognize and analyze emotion data in real time.

[2036] Explanation of program processing

[2037] The server receives personal and emotional data sent from the user's device. Data collection is performed using the smartphone's built-in microphone and camera, which capture voice and facial expression data. The received data is preprocessed on the server, and a user model is generated using a machine learning algorithm. This user model is a digital profile that reflects the user's past behavior and preferences. A digital clone is then generated based on this user model and sent to the user's device.

[2038] The emotion engine runs on the device and recognizes the user's real-time emotional state. The recognized emotional data is analyzed and suggestions appropriate to the user's state are made. For example, in a food delivery application, if a user is feeling stressed, the engine will suggest dishes that are suitable for relieving stress based on past preferences.

[2039] Usage example

[2040] Below are some specific examples of food delivery:

[2041] 1. The user types into the terminal, "I'm feeling stressed, so please give me some suggestions."

[2042] 2. The emotion engine detects the user's emotions in real time and recognizes "stress."

[2043] 3. The server analyzes the user's past order history and preferences based on the user model, and suggests dishes suitable for relieving stress.

[2044] 4. The device displays the suggested dishes to the user, and the user confirms the order.

[2045] 5. The server sends the order to the delivery service and manages the delivery status in real time.

[2046] Prompt Sentence Examples

[2047] An example of a prompt sentence for suggesting a dish to a user who is feeling stressed is as follows:

[2048] For example: "The user is currently feeling stressed. Their past ordering history shows that they like chocolate cake and comfort food to relieve stress. Please recommend appropriate dishes based on their current emotional state."

[2049] As described above, the system of the present invention can provide a fast and accurate response that takes into account the user's emotional state, which can increase user satisfaction, especially in the food delivery field. It can also provide high-level support using emotional data for other online tasks.

[2050] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2051] Step 1:

[2052] User collects data

[2053] Users use their smartphone's camera and microphone to collect personal and emotional data. Input data includes voice and facial expression data. This data is temporarily stored on the device. The collected voice and facial expression data is provided as output.

[2054] Step 2:

[2055] The device sends the data to the server

[2056] The device encrypts the collected personal data and emotion data and transmits them to the server via a secure protocol. The input data includes the collected voice and facial expression data. The output data includes the personal data and emotion data received by the server.

[2057] Step 3:

[2058] The server preprocesses the data

[2059] The server preprocesses the received data, removing noise and normalizing the data. The input data includes the server's personal data and emotion data sent from the device. After preprocessing, the cleaned data is stored on the server as the output.

[2060] Step 4:

[2061] The server generates the user model

[2062] The server uses the preprocessed data to generate a user model based on machine learning algorithms. The input data includes the preprocessed personal data and emotional data. The output is a user model, which is a digital profile that reflects the user's behavioral patterns and preferences.

[2063] Step 5:

[2064] The server creates a digital clone and sends it to the device.

[2065] The server creates a digital clone based on the generated user model and sends it to the terminal. The input data includes the user model. The output data includes the digital clone received by the terminal.

[2066] Step 6:

[2067] Emotion engine recognizes emotional data in real time

[2068] The emotion engine of the device recognizes the user's emotion data in real time. The input data includes real-time emotion data obtained from the user's voice and facial expressions. The output is the detected user's emotional state provided to the device.

[2069] Step 7:

[2070] The emotion engine analyzes the user's emotional state and makes appropriate suggestions to them.

[2071] The emotion engine on the device analyzes the recognized emotion data and the user model and makes suggestions appropriate to the user's state. For example, if a user is feeling stressed, it will suggest dishes that are good for relieving stress. The input data includes the detected emotional state and the user model. The output is a display of the suggestions made to the user.

[2072] Step 8:

[2073] The user confirms the order

[2074] The user confirms the order based on the suggestions displayed on the terminal. The input data includes the suggestions displayed on the terminal. The output is the confirmed order sent to the server.

[2075] Step 9:

[2076] The server processes the order and sends it to the delivery service

[2077] The server processes the user's confirmed order and sends it to the delivery service. The input data includes the user's confirmed order. The output data includes the order details received by the delivery service.

[2078] Step 10:

[2079] The server manages the delivery status

[2080] The server tracks the delivery status from the delivery service in real time and provides the information to the user. The input data includes delivery progress information from the delivery service. The output data includes delivery progress information provided to the user.

[2081] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2082] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2083] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2084] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2085] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2086] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2087] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2088] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2089] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2090] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2091] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2092] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2093] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2094] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2095] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2096] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit),...

Claims

1. The means by which personal data is collected; means for transmitting the collected data to a server; a means for preprocessing the received data by the server and generating a user model using a machine learning algorithm; A means for creating a digital clone using the generated user model and transmitting the digital clone to a terminal; means for the digital clone to perform online tasks in accordance with the user's instructions; A system including:

2. A digital clone can join online meetings and convert speech into text in real time to create minutes. A means for reporting the created minutes to the user is provided. The system of claim 1 .

3. It analyzes the user's schedule, lists high-priority tasks, and provides a means to set reminders. The system of claim 1 .

Images