System

Abstract

To provide a system.SOLUTION: A system includes: means for acquiring feature data representing life log data on a person, a sentence about the person, and a voiceprint of the person; and means for inputting a message input from a user to a generative AI trained on the basis of the feature data to output data output from the generative AI.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Death is an inevitable reality, resulting in people losing communication with their loved ones forever. Even during life, busy schedules or absences can limit communication with important people. [Means for solving the problem]

[0005] This invention combines lifelog data from messenger apps, internet search tools, and other sources with characteristic data such as text and voiceprints, and uses this data as training data to enable AI to recreate the user's memories and personality. It then generates an avatar of the individual on the cloud, which can respond to chats and consultations with acquaintances, friends, and family when the individual is absent or busy during life, and remains an eternal presence after death. Furthermore, the system offers a monthly subscription system in which the degree of reproduction improves the longer the subscription is maintained. This allows people to maintain communication with important people, both during and after death. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the disclosed technique will be described below with reference to the accompanying drawings.

[0008] do.

[0009] First, the terms used in the following description will be explained.

[0010] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Also, the processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit),

[0011] Examples include a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g.,

[0014] hard disk), or magnetic tape.

[0015] In the following embodiments, the coded communication I / F (Interface) refers to a communication processor and

[0016] The communication I / F is an interface that includes a network interface (NIC) and an antenna. The communication I / F controls communication between multiple computers. Examples of communication standards that can be applied to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0017] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0018] [First embodiment]

[0019] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0020] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0021] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0022] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0023] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0024] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0025] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0026] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0027] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0028] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0029] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0030] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0031] "Example 1"

[0032] In one embodiment of the present invention, life log data and feature data such as sentences and voiceprints are collected from messenger apps, internet search tools, and the like that users use on a daily basis.

[0033] This data reflects the user's behavioral patterns, hobbies, interests, personality, etc. By using this data as training data and training the AI, the user's memories and personality can be recreated.

[0034] "Example 2"

[0035] Next, an avatar of the user is generated on the cloud. This avatar is designed to maintain communication with acquaintances, friends, and family when the user is absent, busy, or even after the user has passed away. Specifically, the avatar mimics the user's speech and thought patterns, recreating how the user normally interacts.

[0036] "Example 3"

[0037] Furthermore, the system of the present invention employs a monthly subscription system. The longer a user continues to pay, the more data the system collects and the more the AI ​​learns. This allows the avatar to be more accurately recreated, and the user's memories and personality to be more accurately reproduced.

[0038] The processing flow of each embodiment will be described below.

[0039] "Example 1"

[0040] Step 1: Collect life log data and characteristic data such as text and voiceprints from messenger apps, internet search tools, etc. that users use on a daily basis.

[0041] Step 2: The collected data is used as training data for the AI ​​to learn. Through this learning, the AI ​​will understand the user's behavioral patterns, hobbies, interests, personality, etc.

[0042] Step 3: As the AI ​​learns, it becomes better at replicating the user's memories and personality.

[0043] "Example 2"

[0044] Step 1: Create an avatar of the user on the cloud. This avatar will maintain communication with acquaintances, friends, and family when the user is absent, busy, or even after the user has passed away.

[0045] Step 2: The avatar mimics the user's language and thought patterns, replicating how the user normally interacts.

[0046] "Example 3"

[0047] Step 1: The system of the present invention employs a monthly billing system.

[0048] Step 2: The longer the user continues to pay, the more data the system collects and the more the AI ​​learns.

[0049] Step 3: This increases the fidelity of the avatar, allowing it to more accurately recreate the user's memories and personality.

[0050] Example 1

[0051] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0052] Modern society demands technology that can collect data reflecting a user's behavioral patterns, interests, and personality, and then use that data to recreate their memories and personality. However, existing systems lack integration between the steps of data collection, preprocessing, storage, learning, evaluation, and generation, making it difficult to efficiently recreate a user's memories and personality. Furthermore, there is a lack of ways for users to maintain communication with acquaintances, friends, and family, even when they are absent or have passed away. Furthermore, there is no established sustainable billing model for improving the level of recreation.

[0053] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0054] In this invention, the server includes: means for collecting life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc.; means for preprocessing the collected data and converting it into a format suitable for learning an AI model; means for saving the preprocessed data in a database; means for inputting the saved data as training data into an AI model and training it to reproduce the user's memories and personality; means for evaluating the AI ​​model after training and confirming its performance; means for saving well-rated models; and means for generating an avatar of the user on the cloud. This allows for efficient collection and processing of data reflecting the user's behavioral patterns, interests, and personality, enabling highly accurate reproduction of the user's memories and personality. Furthermore, the system allows the user to maintain communication with acquaintances, friends, and family even when they are absent or after their death, and the level of reproduction can be improved through a sustainable billing model.

[0055] A "messenger app" is a software application that enables users to send and receive text messages, voice messages, images, videos, etc.

[0056] An "Internet search tool" is a software application that allows a user to search for information on the Internet.

[0057] "Life log data" is data that records a user's actions and activities in their daily life.

[0058] "Characteristic data" is data that reflects personal characteristics, such as a user's writing style or voiceprint.

[0059] "Preprocessing" is the process of converting collected data into a format suitable for learning AI models.

[0060] A "database" is a system for efficiently storing, managing, and retrieving data.

[0061] "Training data" is data with correct labels that is used to train an AI model.

[0062] An "AI model" is a mathematical model that uses artificial intelligence technology to analyze data and make predictions and classifications.

[0063] "Learning" is the process by which an AI model discovers patterns and rules based on training data.

[0064] "Evaluation" is the process of checking the performance of an AI model after it has completed training.

[0065] The "cloud" is a collection of computer resources and services provided over the Internet.

[0066] An "avatar" is a digital being that recreates the user's memories and personality.

[0067] A "charging model" is a business model for collecting fees for the use of a service.

[0068] MODE FOR CARRYING OUT THE INVENTION

[0069] This invention is a system that collects life log data and characteristic data such as sentences and voiceprints from messenger apps and internet search tools that users use on a daily basis, and recreates the user's memories and personality based on this data. Specific embodiments of this system are described below.

[0070] Data collection

[0071] The server collects life log data and characteristic data such as sentences and voiceprints from the messenger apps and internet search tools that users use on a daily basis. This data is collected using common software such as messenger apps and internet search tools. The collected data reflects the user's behavioral patterns, hobbies, interests, personality, etc.

[0072] Data Preprocessing

[0073] The server preprocesses the collected data and converts it into a format suitable for learning AI models. Specifically, it tokenizes text data and converts audio data into a spectrogram. This is done using Python libraries such as NLTK and Librosa. For example, NLTK is used to tokenize text data, and Librosa is used to convert audio data into a spectrogram.

[0074] Data storage

[0075] The server saves the preprocessed data to a database, for example, using a database system such as MySQL (registered trademark) or MongoDB, by establishing a database connection and inserting the preprocessed data into the appropriate tables and collections.

[0076] Training an AI model

[0077] The server inputs the preprocessed data into the AI ​​model as training data, and trains it to reproduce the user's memories and personality. This training uses a generative AI model such as GPT-4 (registered trademark) or BERT. Specifically, the server initializes the model, batch processes the data, and executes the training.

[0078] Evaluating and saving the model

[0079] The server evaluates the AI ​​model after training and checks its performance. Evaluation uses metrics such as precision and recall. Models that pass the evaluation are stored in a file system or cloud storage.

[0080] Avatar generation on the cloud

[0081] The server uses the well-reviewed model to generate a cloud-based avatar of the user, which recreates the user's memories and personality and is used to maintain communication with acquaintances, friends, and family.

[0082] Specific examples

[0083] Example 1: Data collection from messenger apps

[0084] The server collects messages sent and received by users on popular messaging apps. These messages reflect the user's conversation patterns and interests. This message data is preprocessed and trained into an AI model to reproduce the user's conversation style.

[0085] Example 2: Data collection from internet search tools

[0086] The server collects search queries performed by users using common internet search tools. The collected search queries reflect the user's interests and concerns. This search data is preprocessed and trained into an AI model to reproduce the user's interests.

[0087] Prompt Sentence Examples

[0088] "Create a program to collect messages sent by users on popular messaging apps and train an AI model to replicate the user's conversational style."

[0089] "You will create a program that collects search queries performed by users on popular internet search tools and trains an AI model to replicate the user's interests."

[0090] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0091] Step 1: Data collection

[0092] The server collects life log data and feature data such as sentences and voiceprints from the messenger apps and internet search tools that users use on a daily basis. It receives the log data from the messenger apps and internet search tools as input and generates the collected raw data as output. Specifically, the server obtains data using APIs and scraping technology, and collects the data using secure communication protocols (e.g., HTTPS).

[0093] Step 2: Data Preprocessing

[0094] The server preprocesses the collected data and converts it into a format suitable for training the AI ​​model. It receives the collected raw data as input and produces preprocessed data as output. Specifically, the server performs the following operations:

[0095] Tokenizing text data: Splitting sentences into words and phrases using Python's NLTK library.

[0096] Spectrogram conversion of audio data: Use the Librosa library in Python to convert audio data into a frequency spectrum.

[0097] Step 3: Save your data

[0098] The server stores the preprocessed data in a database. It receives the preprocessed data as input and generates the data stored in the database as output. Specifically, the server performs the following operations:

[0099] Establishing a database connection: Connect to MySQL or MongoDB.

[0100] Inserting Data: Inserting the pre-processed data into the appropriate tables and collections.

[0101] Step 4: Training the AI ​​model

[0102] The server inputs the preprocessed data as training data into the AI ​​model, which then learns to replicate the user's memories and personality. As input, it receives the preprocessed data stored in the database, and as output, it generates a trained AI model. Specifically, the server performs the following operations:

[0103] Model initialization: Initialize a generative AI model such as GPT-4 or BERT.

[0104] Data Batching: Preprocessed data is fed into the model in batches.

[0105] Execute training: Input data into the model and execute training.

[0106] Step 5: Evaluate and save the model

[0107] The server evaluates the AI ​​model after training and checks its performance. As input, it receives the trained AI model and evaluation data, and as output, it generates the evaluation results and a saved model. Specifically, the server performs the following operations:

[0108] Model evaluation: Calculating metrics such as precision and recall.

[0109] Save the model: Save the model with a good evaluation to the file system or cloud storage.

[0110] Step 6: Cloud avatar generation

[0111] The server generates an avatar of the person on the cloud using the model with a good evaluation. It receives the stored AI model as input and generates the avatar on the cloud as output. Specifically, the server performs the following operations:

[0112] Deploying the model: Deploying the AI ​​model to a cloud environment.

[0113] Avatar Generation: Using the deployed model, an avatar is generated that recreates the user's memories and personality.

[0114] (Application example 1)

[0115] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0116] In modern society, there is a demand for providing optimal content for each individual user by utilizing their life log data and characteristic data. However, conventional systems have had difficulty recreating a user's memories and personality and recommending personalized content. Furthermore, there has been a lack of ways for users to maintain communication with acquaintances and family when they are absent, busy, or even after death. A new system is needed to solve these issues.

[0117] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0118] In this invention, the server includes a means for linking life log data from messenger apps, internet search tools, etc. with feature data such as text and voiceprints; a means for AI to recreate the user's memories and personality using the data as training data; a means for generating an avatar of the user on the cloud; and a means for automatically recommending content based on the user's preferences and interests based on the user's life log data and feature data. This makes it possible to recreate the user's memories and personality and provide personalized content. It also allows the user to maintain communication with acquaintances and family even when they are away, busy, or even after death.

[0119] A "messenger app" is an application that allows users to send and receive text messages, voice messages, images, videos, etc.

[0120] An "Internet search tool" is software or a web service that allows users to search for information on the Internet.

[0121] "Life log data" is data recorded about a user's daily life behavior and activities.

[0122] "Characteristic data" is data used to identify individual users, such as their writing or voiceprint.

[0123] "Training data" is a dataset used for AI learning.

[0124] "AI" stands for artificial intelligence, a technology that allows machines to mimic human intelligence.

[0125] "Reproducing the user's memories and personality" means that the AI ​​imitates the user's thinking and behavior patterns based on the user's past actions and statements.

[0126] "Creating a digital avatar on the cloud" means using cloud computing technology to create a digital avatar of a user.

[0127] "Automatic content recommendation" means that AI suggests appropriate information and entertainment based on a user's preferences and interests.

[0128] The system for implementing this invention collects user life log data and characteristic data, recreates the user's memories and personality based on the collected data, and recommends personalized content. Specific embodiments of this system are described below.

[0129] System Configuration

[0130] Hardware

[0131] Server: A server for collecting, processing, and storing data, as well as training and inferencing AI models.

[0132] Device: The device used by the user, such as a smartphone or computer.

[0133] software

[0134] Messenger app: An application that allows users to send and receive text messages, voice messages, images, videos, etc.

[0135] Internet Search Tools: Software or web services that allow users to search for information on the Internet.

[0136] AI model: A generative AI model that recreates a user's memories and personality based on their life log data and feature data, and recommends personalized content.

[0137] Data collection and processing

[0138] The server collects user life log data and feature data from messaging apps and internet search tools, including user message history, search history, and voice data. The collected data is preprocessed and then used as training data for the AI ​​model.

[0139] AI model training and inference

[0140] The server trains the AI ​​model based on the collected data. Specifically, text data is digitized using TfidfVectorizer and a clustering algorithm (e.g., KMeans) is used to create a user profile. Voice data is converted to text using speech recognition technology and processed in the same way.

[0141] Content Recommendations

[0142] The server uses a trained AI model to automatically recommend content based on the user's preferences and interests, including entertainment content such as movies, music, and articles. The recommended content is then sent to the user's device.

[0143] Specific examples

[0144] For example, if User A frequently talks about "movies" on a messaging app, the server collects that data and trains the AI ​​model. If User A's search history frequently includes searches for "latest movie reviews" and "movie trailers," the server creates a profile for User A based on this data. As a result, User A is recommended the latest movie review articles and movie trailer videos.

[0145] Prompt Sentence Examples

[0146] Develop an application that recommends content based on the user's interests and preferences, based on data collected from the user's messaging app and search history. Specifically, implement a feature that analyzes the topics and keywords frequently discussed by the user and automatically recommends content such as movies, music, and articles.

[0147] In this way, a system can be constructed that utilizes users' life log data and characteristic data to deliver personalized content.

[0148] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0149] Step 1:

[0150] The server collects user life log data and feature data from messenger apps and internet search tools, specifically, user message history, search history, voice data, etc. This data is collected using APIs and database queries.

[0151] Input: Raw data from messaging apps and search tools

[0152] Output: Collected lifelog data and feature data

[0153] Step 2:

[0154] The server preprocesses the collected data: text data is digitized using TfidfVectorizer, and voice data is converted to text using speech recognition technology, converting the data into a format suitable for the AI ​​model.

[0155] Input: Collected lifelog data and feature data

[0156] Output: Preprocessed data

[0157] Step 3:

[0158] The server trains the AI ​​model based on the preprocessed data. Specifically, it creates a user profile using a clustering algorithm (e.g., KMeans), which allows the model to reflect the user's behavioral patterns and interests.

[0159] Input: Preprocessed data

[0160] Output: A trained AI model

[0161] Step 4:

[0162] The server uses a trained AI model to automatically recommend content based on the user's preferences and interests, specifically selecting entertainment content such as movies, music, and articles based on the user profile.

[0163] Input: Trained AI model, user profile

[0164] Output: Recommended content

[0165] Step 5:

[0166] The server notifies the user of the recommended content via push notifications or in-app messages.

[0167] Input: Recommended content

[0168] Output: Content displayed on the user's device

[0169] In this way, a system is constructed that utilizes users' life log data and characteristic data to deliver personalized content.

[0170] Example 2

[0171] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0172] In modern society, there is a need to maintain communication with acquaintances, friends, and family even when users are away, busy, or have passed away. However, conventional technologies have struggled to accurately mimic a user's language and thought patterns to achieve natural dialogue. Furthermore, there has been a lack of means to effectively utilize a user's past messages and conversation history to train a generative AI model. This has led to the problem of user avatars generating unnatural responses.

[0173] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0174] In this invention, the server includes: means for linking life log data and feature data such as sentences and voiceprints from messenger apps, internet search tools, etc., means for a generative AI model to reproduce the user's memories and personality using the linked data as training data; means for generating an avatar of the user on the cloud; means for collecting the user's past messages and dialogue history and storing them in a database; means for preprocessing the collected data, cleaning the text, tokenizing, and extracting important phrases and patterns; means for training the generative AI model using the preprocessed data; means for generating an avatar of the user using the trained generative AI model; means for receiving messages from acquaintances and family members, inputting them as prompts into the generative AI model, and generating appropriate replies; and means for sending the generated replies to acquaintances and family members. This enables the system to accurately mimic the user's language and thought patterns and realize natural conversations.

[0175] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[0176] An "Internet search tool" is software that allows users to search for and retrieve information on the Internet.

[0177] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, and health data.

[0178] "Characteristic data" is data that represents a user's individual characteristics, including writing style and voiceprint.

[0179] A "generative AI model" is a model that uses artificial intelligence technology to imitate a user's language usage and thought patterns.

[0180] The "cloud" is a collection of computer resources and services provided over the Internet.

[0181] An "avatar" is a virtual entity that mimics the user's speech and thought patterns and converses on the user's behalf.

[0182] A "database" is a system for efficiently storing, searching, and managing data.

[0183] "Preprocessing" refers to the process of converting data into a format suitable for analysis and model training.

[0184] "Tokenization" is the process of dividing text data into smaller units such as words or phrases.

[0185] A "prompt" is text that is input to a generative AI model and contains instructions for the model to generate an appropriate response.

[0186] This invention is a system for maintaining communication with acquaintances, friends, and family even when the user is away, busy, or has passed away. This system links life log data from messenger apps, internet search tools, etc. with feature data such as text and voiceprints, and uses this data as training data to generate an AI model that reproduces the user's memories and personality. A specific embodiment of this system is described below.

[0187] 1. Collection of User Data

[0188] The server collects the user's past messages and conversation history. This includes emails, chat logs, and social media posts. The server stores this data in a database. For example, the server accesses the user's email account and downloads past emails. It also uses the chat app's API to retrieve the user's chat log.

[0189] 2. Data Preprocessing

[0190] The server pre-processes the collected data, which includes cleaning the text, tokenizing it, and extracting important phrases and patterns. Specifically, the server removes unnecessary HTML tags and special characters from the text, splits the text into words and phrases (tokenization), and extracts frequently occurring phrases and patterns, which are then stored in a database.

[0191] 3. Training the generative AI model

[0192] The server uses the preprocessed data to train a generative AI model. For example, OpenAI's GPT-4 is used here. The server inputs the preprocessed data into the generative AI model and trains the model. Once trained, the model will be able to mimic the user's language and thought patterns.

[0193] 4. The Creation of an Avatar

[0194] The server uses a trained generative AI model to generate an avatar for the user, which mimics the user's speech and thought patterns to replicate how the user normally interacts. The generated avatar is then stored in a database.

[0195] 5. Receiving messages and generating replies

[0196] The device receives messages from acquaintances and family members. The device inputs this message as a prompt into the generative AI model to generate an appropriate reply. For example, if a message is received from an acquaintance saying, "What are your plans for tomorrow?", the device inputs this message into the generative AI model as a prompt. An example of a prompt is "User's name: Yamada Taro, User's past message: 'I'm busy today, so please check my plans for tomorrow,' Prompt: 'When Yamada Taro is busy, please generate a reply to check tomorrow's plans.'"

[0197] 6. Sending a Reply

[0198] The device then sends the generated reply to a friend or family member. For example, it receives a reply from the generative AI model saying, "Tomorrow's schedule is a meeting at 10:00 AM and a client meeting at 2:00 PM," and sends it to a friend or family member.

[0199] In this way, the server and device work together to generate an avatar of the user, enabling them to maintain communication with acquaintances, friends, and family even when they are absent, busy, or have passed away.

[0200] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0201] Step 1: Collect user data

[0202] The server collects the user's past messages and conversation history. Input includes emails, chat logs, and social media posts. The server stores this data in a database. Specifically, the server accesses the email account and downloads past emails. It also uses the chat app's API to retrieve the user's chat log. The collected data is stored in the database as output.

[0203] Step 2: Preprocessing the data

[0204] The server preprocesses the collected data. The input includes the data collected in step 1. The server cleans the text, tokenizes it, and extracts important phrases and patterns. Specifically, the server removes unnecessary HTML tags and special characters from the text, and splits the text into words and phrases (tokenization). It then extracts frequently occurring phrases and patterns and stores them in a database. The output is the preprocessed data.

[0205] Step 3: Training the generative AI model

[0206] The server trains a generative AI model using the preprocessed data. The input includes the data preprocessed in step 2. The server inputs the preprocessed data into the generative AI model and trains the model. Specifically, the server inputs the data into the generative AI model (e.g., GPT-4) and adjusts the model parameters. The output is a fully trained generative AI model.

[0207] Step 4: Creating an Avatar

[0208] The server uses the trained generative AI model to generate an avatar for the user. The input includes the generative AI model trained in step 3. The server inputs the user's characteristics into the generative AI model to generate an avatar. Specifically, the server inputs the user's language and thought patterns into the model to generate an avatar. As an output, the generated avatar is stored in a database.

[0209] Step 5: Receiving the message and generating a reply

[0210] The device receives messages from acquaintances and family members. The input includes messages from acquaintances and family members. The device inputs these messages as prompts to the generative AI model, which then generates an appropriate reply. Specifically, the device uses the messaging app's API to obtain the received message and inputs it as a prompt to the generative AI model. The output is a reply from the generative AI model.

[0211] Step 6: Send your reply

[0212] The device sends the generated reply to the friend or family member. The input includes the reply generated in step 5. Specifically, the device uses the messaging app's API to send the generated reply. The output is the reply sent to the friend or family member.

[0213] In this way, the server and device work together to generate an avatar of the user, enabling them to maintain communication with acquaintances, friends, and family even when they are absent, busy, or have passed away.

[0214] (Application example 2)

[0215] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0216] In modern society, it is often difficult to provide customer service when users are absent or busy. Furthermore, there are limited ways to maintain communication with acquaintances, friends, and family members even after a user passes away. Furthermore, when providing customer service in a virtual store, it is necessary to mimic the user's language and thought patterns, but there is a lack of technology to achieve this.

[0217] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc.; means for AI to reproduce the user's memories and personality using the data as training data; means for generating an avatar of the user on the cloud; means for the user's avatar to interact with customers in a virtual store; means for generating answers to customer questions using a generative AI model; means for imitating the user's language and thought patterns using prompt sentences; and means for installation on a smartphone or head-mounted display. This enables customer interaction even when the user is absent or busy, and allows communication with acquaintances, friends, and family to be maintained even after the user's death. Furthermore, customer interaction in a virtual store that imitates the user's language and thought patterns is possible.

[0218] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[0219] An "Internet search tool" is software that searches for information on the Internet and provides it to users.

[0220] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, health data, etc.

[0221] "Characteristic data" is data used to identify an individual, such as a user's writing or voiceprint.

[0222] "Training data" is data used to train a machine learning model.

[0223] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[0224] The "cloud" is a collection of computer resources and services provided over the Internet.

[0225] An "avatar" is a virtual being created to mimic the user's speech and thought patterns.

[0226] A "virtual store" is a virtual store that exists on the Internet, where users can purchase goods and services.

[0227] "Customer service" refers to responding to and dealing with questions and requests from customers.

[0228] A "generative AI model" is an artificial intelligence model that generates text and speech by imitating a user's language and thought patterns.

[0229] A "prompt" is text that is input to a generative AI model, instructing the model how to respond.

[0230] A "smartphone" is a mobile phone that can connect to the Internet and run applications.

[0231] A "head-mounted display" is a display device worn on the head that provides virtual reality and augmented reality experiences.

[0232] The system for implementing this invention collects user life log data and feature data, and generates an avatar of the user on the cloud based on the collected data. A specific embodiment of this system will be described below.

[0233] First, the server collects the user's life log data and feature data from messenger apps, internet search tools, etc. Life log data includes the user's daily behavior history, location information, health data, etc. Feature data includes data that can be used to identify an individual, such as the user's writing style and voiceprint.

[0234] The server then uses this data as training data to train a generative AI model, which is then used to mimic the user's language and thought patterns. Specifically, it uses prompt sentences to mimic the user's language and thought patterns.

[0235] The server generates an avatar of the user in the cloud. This avatar exists to maintain communication with acquaintances, friends, and family even when the user is absent, busy, or even after the user passes away. The avatar can respond to customers in a virtual store. For example, if a customer asks, "What are the features of this product?", the avatar uses a generative AI model to generate an appropriate answer.

[0236] This system is implemented as an application installed on a smartphone or head-mounted display, and can improve the customer experience by allowing the avatar to respond to customers even when the user is absent or busy.

[0237] As a specific example, by using the following prompt sentence, the avatar can respond to customers by imitating the user's language and thought patterns.

[0238] Example prompt sentence:

[0239] You are a salesperson in a virtual store. Please answer the following questions by imitating the user's language and thought patterns.

[0240] Question: {question}

[0241] In this way, the avatar can respond to customers even when the user is absent or busy, and can maintain communication with acquaintances, friends, and family even after the user's death.It is also possible for the avatar to respond to customers in a virtual store by imitating the user's language and thought patterns.

[0242] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0243] Step 1:

[0244] The server collects users' life log data and feature data from messenger apps and internet search tools. The input includes the user's daily behavior history, location information, health data, text, voiceprint, etc. This data is collected and stored in a database. The output is the collected life log data and feature data.

[0245] Step 2:

[0246] The server preprocesses the collected lifelog data and feature data. The data collected in step 1 is used as input. Data processing such as data cleaning, normalization, and feature extraction is performed, and the data is converted into a format suitable for training a generative AI model. The preprocessed data is obtained as output.

[0247] Step 3:

[0248] The server uses the preprocessed data as training data to train the generative AI model. The preprocessed data from step 2 is used as input. The generative AI model analyzes the data and extracts patterns to learn the user's language and thought patterns. The output is a trained generative AI model.

[0249] Step 4:

[0250] The server generates an avatar for the user on the cloud. The generative AI model trained in step 3 is used as input. Based on the generative AI model, a virtual being is generated that mimics the user's speech and thought patterns. The output is an avatar generated on the cloud.

[0251] Step 5:

[0252] The device runs an application that allows the user's avatar to interact with customers in a virtual store. The input includes the avatar generated on the cloud and the customer's question. The device sends the customer's question to the cloud and generates an appropriate answer using a generative AI model. The output is the answer for the customer.

[0253] Step 6:

[0254] The server uses prompts to mimic the user's language and thought patterns. The input includes the customer's question and the prompt. The generative AI model mimics the user's language and thought patterns based on the prompt and generates an appropriate answer. The output is the mimicked answer.

[0255] Step 7:

[0256] The terminal displays the answer to the customer through an application installed on a smartphone or head-mounted display. It includes the answer generated in step 6 as input. The terminal displays the answer to the customer, improving the customer experience. The output is the answer displayed to the customer.

[0257] Example 3

[0258] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0259] In modern society, there is a demand for digitizing a user's memories and personality and permanently storing and reproducing them. However, conventional systems have difficulty accurately reproducing a user's memories and personality, and lack a mechanism for improving the degree of reproduction as the user uses the system over a long period of time. For this reason, there is a need for a system that can reproduce a user's memories and personality with high accuracy and further improve the accuracy with long-term use.

[0260] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0261] In this invention, the server includes means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., means for AI to reproduce the user's memories and personality using the linked data as training data, means for generating an avatar of the user on the cloud, means for collecting data provided by the user and storing it in a database, means for preprocessing the collected data and converting it into a format suitable for the generative AI model, means for training the generative AI model using the preprocessed data, means for receiving prompts from the user and generating responses using the generative AI model, and means for providing the generated responses to the user. This makes it possible to reproduce the user's memories and personality with high accuracy, and to further improve this accuracy with long-term use.

[0262] A "messenger app" is software that allows users to send and receive text and voice messages.

[0263] An "Internet search tool" is software that allows users to search for information on the Internet.

[0264] "Life log data" refers to data related to a user's daily life, including behavioral history and activity records.

[0265] "Characteristic data" is data that indicates the individual characteristics of a user, and includes sentences, voiceprints, etc.

[0266] "Training data" is data used to train a generative AI model, and is a dataset that has been assigned a correct answer label.

[0267] A "generative AI model" is an artificial intelligence model used to recreate a user's memories and personality.

[0268] The "cloud" is a collection of computer resources and services provided over the Internet.

[0269] An "avatar" is a digital being that recreates the user's memories and personality.

[0270] A "database" is a system for systematically storing and managing collected data.

[0271] "Preprocessing" refers to the process of converting collected data into a format suitable for a generative AI model.

[0272] A "prompt" is a question or request that a user enters into a system.

[0273] A "response" is the answer that a generative AI model generates in response to a prompt.

[0274] This invention is a system that digitizes a user's memories and personality and generates an avatar on the cloud. The system collects life log data and feature data from messenger apps, internet search tools, etc., and uses this data as training data for a generative AI model. A specific embodiment of this system is described below.

[0275] First, users provide life log data such as text messages, voice data, and image data through messenger apps or internet search tools. The device then sends this data to a server, which then stores the received data in a database.

[0276] The server then preprocesses the collected data. Specifically, text data is tokenized using natural language processing (NLP) technology, and voice data is converted to text using voice recognition software. Image data undergoes feature extraction using image recognition technology. Machine learning frameworks such as TENSORFLOW® and PyTorch are used for this preprocessing.

[0277] The preprocessed data is used to train the generative AI model. The server uses this data to train the generative AI model, teaching it patterns to reproduce the user's memories and personality. As the learning progresses, the model's reproducibility improves.

[0278] When a user inputs a question or request into the system as a prompt, the device sends the prompt to the server, which uses a generative AI model to generate an appropriate response to the prompt, which is then sent to the device and provided to the user.

[0279] For example, if a user asks, "What is my favorite movie?", the server uses a generative AI model to search for information about "favorite movies" from the user's past data and generate a response such as, "Your favorite movie is Inception." Similarly, if a user asks, "Where was the last place I traveled?", the server generates a response such as, "You went to Hawaii last summer."

[0280] This system employs a monthly subscription system, and the longer a user continues to pay, the more data is collected and the more the AI ​​learns. This improves the reproducibility of the avatar, making it possible to more accurately reproduce the user's memories and personality. The flow of the identification process in Example 3 will be explained using Figure 15.

[0281] Step 1:

[0282] Users provide lifelog data such as text messages, voice data, and image data through messenger apps and internet search tools. The device sends this data to the server. The input is the lifelog data from the user, and the output is the data sent to the server.

[0283] Step 2:

[0284] The server stores the received data in a database. Specifically, it stores text messages, voice data, and image data in the corresponding database tables. The input is the data sent from the terminal, and the output is the data stored in the database.

[0285] Step 3:

[0286] The server preprocesses the collected data. Text data is tokenized using natural language processing (NLP) techniques to extract important keywords. Voice data is converted to text using voice recognition software. Image data is subjected to feature extraction using image recognition techniques. The input is raw data stored in a database, and the output is preprocessed data.

[0287] Step 4:

[0288] The server uses the preprocessed data to train a generative AI model. Specifically, it uses machine learning frameworks such as TensorFlow and PyTorch to train the model to learn patterns that will reproduce the user's memories and personality. The input is the preprocessed data, and the output is a trained generative AI model.

[0289] Step 5:

[0290] The user inputs a question or request to the system as a prompt. The terminal sends this prompt to the server. The input is the prompt from the user, and the output is the prompt sent to the server.

[0291] Step 6:

[0292] The server inputs the received prompt into the generative AI model to generate an appropriate response. The generative AI model generates the most appropriate answer based on the user's past data. The input is the prompt and the trained generative AI model, and the output is the generated response.

[0293] Step 7:

[0294] The server sends the generated response to the terminal, which displays this response to the user. The input is the generated response, and the output is the response provided to the user.

[0295] Specifically, when a user asks, "What's my favorite movie?", the server uses the generative AI model to search for information about "favorite movies" from the user's past data and generates a response such as, "Your favorite movie is Inception." Similarly, when a user asks, "Where was the last place I traveled?", the server generates a response such as, "You went to Hawaii last summer."

[0296] (Application example 3)

[0297] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0298] In conventional systems, the accuracy of the avatars that reproduce the user's memories and personality was low, making it difficult to recommend individually customized content. Furthermore, there was a lack of a mechanism for improving the accuracy of the system as the user uses it over a long period of time, making it difficult to improve the user experience.

[0299] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for linking life log data and feature data such as sentences and voiceprints from messenger apps, internet search tools, etc., a means for AI to reproduce the user's memories and personality using the data as training data, a means for generating an avatar of the user on the cloud, and a means for collecting user data, generating an avatar using an AI model, and recommending individually customized content. This makes it possible to reproduce the user's memories and personality with high accuracy and recommend individually customized content.

[0300] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[0301] An "Internet search tool" is software that allows users to search for information on the Internet.

[0302] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, health data, etc.

[0303] "Characteristic data" is data used to identify an individual, such as a user's writing or voiceprint.

[0304] "Training data" is data used to train a machine learning model.

[0305] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence.

[0306] "Recreating the user's memories and personality" means imitating the user's past experiences and personality based on collected data.

[0307] The "cloud" is a collection of computer resources and services provided over the Internet.

[0308] An "avatar" is a virtual being that recreates the user's memories and personality.

[0309] "Personalized Content" is information and entertainment specially selected based on a user's preferences and interests.

[0310] An "AI model" is a mathematical model that uses artificial intelligence algorithms to analyze data and make predictions and classifications.

[0311] A "monthly subscription" is a pricing system in which users can use a service by paying a fixed monthly fee.

[0312] A system for implementing this invention is configured as follows: The server has a means for linking life log data with feature data such as text and voiceprints from messenger apps, internet search tools, etc. This makes it possible to collect data about the user's daily life and data for identifying individuals.

[0313] Next, the server has a means for the AI ​​to recreate the user's memories and personality using the collected data as training data. Specifically, based on the collected life log data and feature data, the AI ​​model mimics the user's past experiences and personality. This AI model is implemented using OpenAI's API.

[0314] Furthermore, the server has a means for generating an avatar of the user on the cloud. The generated avatar is a virtual entity that reproduces the user's memories and personality, and is managed on the cloud.

[0315] The server also has a means for collecting user data, generating an avatar using an AI model, and recommending individually customized content. Specifically, the server collects data such as the user's viewing history, preferences, and feedback, and uses the AI ​​model to recommend the most suitable content to the user based on this data. This recommendation provides information and entertainment specially selected based on the user's preferences and interests.

[0316] As users continue to pay over time, the system collects more data and the AI ​​learns more, allowing the avatar to become more realistic and more accurately recreate the user's memories and personality.

[0317] As a concrete example, consider a case where a user likes "action movies" and "comedy movies," likes "movie1," and dislikes "movie2." An example of a prompt sentence that generates an avatar based on this data and recommends content is as follows:

[0318] Example prompt sentence:

[0319] Generate an avatar that recreates the user's memories and personality based on the user's data: {"user_id": "user123", "viewing_history": ["movie1", "movie2"], "preferences": ["action", "comedy"], "feedback": ["liked movie1", "disliked movie2"]}.

[0320] By inputting this prompt into OpenAI's API, an avatar that replicates the user's memories and personality is generated, and the system can then recommend the most appropriate content to the user.

[0321] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0322] Step 1:

[0323] The server collects life log data and feature data from messenger apps and internet search tools. Specifically, it obtains users' text messages, voice messages, search history, location information, etc. The input is various user data, and the output is the collected life log data and feature data.

[0324] Step 2:

[0325] The server inputs the collected life log data and feature data into the AI ​​model as training data. Specifically, it preprocesses this data and converts it into a format that the AI ​​model can learn from. The input is the preprocessed data, and the output is the training data used to train the AI ​​model.

[0326] Step 3:

[0327] The server uses an AI model to generate an avatar that replicates the user's memories and personality. Specifically, it uses OpenAI's API to generate prompts based on the user's data and input them into the AI ​​model. The input is the prompt, and the output is the generated avatar.

[0328] Step 4:

[0329] The server stores the generated avatar on the cloud. Specifically, it uses a cloud storage service to securely store the avatar data. The input is the generated avatar, and the output is the avatar data stored on the cloud.

[0330] Step 5:

[0331] The server collects data such as the user's viewing history, preferences, and feedback, and uses an AI model to recommend individually customized content. Specifically, it generates prompts based on the user's data and inputs them into the AI ​​model. The input is the user's viewing history and preferences, and the output is the recommended content.

[0332] Step 6:

[0333] The user receives the recommended content and views or uses it. Specifically, the user displays and enjoys the recommended content using a smartphone or other device. The input is the recommended content, and the output is the user's viewing or usage history.

[0334] Step 7:

[0335] As users continue to pay over time, the server collects more data and continues to train the AI ​​model. Specifically, the server periodically collects user usage data through a monthly subscription and retrains the AI ​​model. The input is continuously collected user data, and the output is the retrained AI model.

[0336] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0337] "Example 1"

[0338] One embodiment of the present invention is a system incorporating an emotion engine. This system recognizes the user's emotions and adjusts the avatar's response accordingly. Specifically, when the user is happy, the avatar will speak in a bright tone. Conversely, when the user is sad, the avatar will speak in a calm tone. In this way, the emotion engine grasps the user's emotional state in real time and adjusts the avatar's response accordingly.

[0339] "Example 2"

[0340] The emotion engine also controls the avatar's behavior according to the user's emotional state. For example, when the user is angry, the avatar chooses to apologize. When the user is happy, the avatar chooses to congratulate. In this way, the emotion engine understands the user's emotional state and controls the avatar's behavior accordingly.

[0341] "Example 3"

[0342] Furthermore, the emotion engine controls the facial expression of the avatar according to the user's emotional state. For example, when the user is happy, the avatar makes a smiling face. When the user is sad, the avatar makes a sad face. In this way, the emotion engine understands the user's emotional state and controls the facial expression of the avatar accordingly.

[0343] The processing flow of each embodiment will be described below.

[0344] "Example 1"

[0345] Step 1: The emotion engine understands the user's emotional state in real time.

[0346] Step 2: The emotion engine adjusts the avatar's response according to the emotional state it has determined.

[0347] Step 3: The avatar responds with a tailored response to the user.

[0348] "Example 2"

[0349] Step 1: The emotion engine understands the user's emotional state in real time.

[0350] Step 2: The emotion engine controls the avatar's behavior according to the emotional state it has grasped.

[0351] Step 3: The avatar performs the controlled action on the user.

[0352] "Example 3"

[0353] Step 1: The emotion engine understands the user's emotional state in real time.

[0354] Step 2: The emotion engine controls the avatar's facial expression according to the emotional state it has grasped.

[0355] Step 3: The avatar displays a controlled facial expression to the user.

[0356] Example 1

[0357] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0358] In modern society, there is a demand for systems that can accurately grasp a user's behavioral patterns and emotional state and engage in dialogue accordingly. However, conventional systems have had difficulty recognizing a user's emotions in real time and generating appropriate responses accordingly. Furthermore, data collection and analysis to recreate a user's memories and personality were insufficient, making it difficult to increase user satisfaction.

[0359] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0360] In this invention, the server includes means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., means for AI to reproduce the user's memories and personality using this as training data, means for generating an avatar of the user on the cloud, means for recognizing the user's emotions in real time using an emotion engine, and means for adjusting the avatar's tone and dialogue content according to the user's emotional state. This makes it possible to have a dialogue in real time according to the user's emotional state and reproduce the user's memories and personality with high accuracy.

[0361] A "messenger app" is a software application that enables users to send and receive text messages, voice messages, images, videos, etc.

[0362] An "Internet search tool" is a software application that allows a user to search for information on the Internet.

[0363] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, and activity records.

[0364] "Text" refers to text data entered by a user, including messages, comments, posts, etc.

[0365] A "voiceprint" is characteristic data extracted from a user's voice data, and includes voice characteristics that can be used to identify an individual.

[0366] "Feature data" is data that shows specific patterns or characteristics extracted from a user's behavior or comments.

[0367] "Training data" is a dataset used to train a machine learning model, and includes input data and corresponding ground truth data.

[0368] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn, reason, recognize, and make judgments.

[0369] "Recreating the user's memories and personality" means recreating the user's past actions and statements based on collected data, and imitating the user's personality and preferences.

[0370] The "cloud" is a collection of computer resources and services provided over the Internet, and is an infrastructure for storing and processing data.

[0371] An "avatar" is a virtual entity that interacts on behalf of the user and is generated based on the user's memories and personality.

[0372] An "emotion engine" is a software component that analyzes and recognizes emotions from a user's text and voice.

[0373] "Real-time recognition" means analyzing emotions instantly in response to user input and providing results.

[0374] "Tone" refers to the tone and atmosphere of speech or text in a conversation, and is an element used to convey emotion and intention.

[0375] "Adjusting the content of the dialogue" means changing the content and expression of the message sent by the avatar depending on the user's emotional state.

[0376] This invention is a system that collects life log data and characteristic data such as sentences and voiceprints from the messenger apps and internet search tools that users use daily, and recreates the user's memories and personality based on this data.Furthermore, by combining it with an emotion engine, it realizes dialogue that corresponds to the user's emotional state.

[0377] Data collection

[0378] When users use messenger apps (e.g., WhatsApp) or internet search tools (e.g., Google® Search), life log data (e.g., chat history, search history) and characteristic data such as sentences and voiceprints are collected from these applications. To collect the data, a script is used to retrieve the data through APIs.

[0379] Data Preprocessing

[0380] The server receives the collected data and filters out unnecessary information. For example, it removes unnecessary advertising messages from chat history and extracts only user comments. This process is performed using the Python pandas library.

[0381] Data analysis and feature extraction

[0382] The server analyzes the preprocessed data and extracts features such as the user's behavioral patterns, hobbies, interests, and personality. For example, natural language processing (NLP) techniques are used to extract emotions and topics from the user's comments. Libraries such as NLTK and spaCy are used for this analysis.

[0383] Training an AI model

[0384] The server uses the extracted feature data as training data to train a generative AI model (e.g., GPT-4). This gives the model the ability to reproduce the user's memories and personality. Frameworks such as TensorFlow and PyTorch are used for training.

[0385] emotion recognition

[0386] The server uses an emotion engine (e.g., IBM Watson (registered trademark) emotion analysis API) to recognize emotions from the user's text or voice in real time. For example, if a user sends a message saying, "I had so much fun today!", the emotion engine will recognize the emotion "fun."

[0387] Avatar Reaction Adjustment

[0388] The device adjusts the tone and dialogue of the avatar based on information from the emotion engine. For example, if the user is happy, the avatar will respond in a cheerful tone, saying, "That's great! What happened?" Conversely, if the user is sad, the avatar will respond in a calmer tone, saying, "That must be hard. Is there anything I can do to help?"

[0389] Examples and prompts

[0390] Specific examples

[0391] If a user sends a message in a messenger app saying, "I had so much fun today!", the server receives this message and recognizes the emotion "fun" through the emotion engine. Based on this information, the device's avatar responds in a cheerful tone, saying, "That's great! What happened?"

[0392] Prompt Sentence Examples

[0393] "Design a system that analyzes the emotions of messages sent by users on a messenger app and has an avatar respond in an appropriate tone. Specifically, if the user is happy, the avatar should respond in a brighter tone, and if the user is sad, the avatar should respond in a calmer tone."

[0394] In this way, a system is constructed that realizes a dialogue according to the emotional state of the user.

[0395] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0396] Step 1:

[0397] When users use messenger apps or internet search tools, life log data and characteristic data such as sentences and voiceprints are collected from these applications.

[0398] Input: User chat history, search history, and voice data

[0399] Output: Collected life log data, text data, and voiceprint data

[0400] Specific operation: Executes a script that retrieves data through the API and saves it in the database.

[0401] Step 2:

[0402] The server receives the collected data and filters out unnecessary information.

[0403] Input: Collected life log data, text data, voiceprint data

[0404] Output: Preprocessed data (data with unnecessary information removed)

[0405] Specific operation: Using Python's pandas library, create a data frame and remove unnecessary rows and columns.

[0406] Step 3:

[0407] The server analyzes the preprocessed data and extracts characteristics such as the user's behavioral patterns, hobbies, interests, and personality.

[0408] Input: Preprocessed data

[0409] Output: User characteristic data (behavioral patterns, hobbies, interests, personality, etc.)

[0410] What it does: Extract sentiment and topics from text data using natural language processing (NLP) techniques, using libraries such as NLTK and spaCy.

[0411] Step 4:

[0412] The server uses the extracted feature data as training data to train the generative AI model.

[0413] Input: User feature data

[0414] Output: A trained generative AI model

[0415] What it does: Train generative AI models (e.g., GPT-4) using frameworks like TensorFlow and PyTorch.

[0416] Step 5:

[0417] The server uses an emotion engine to recognize emotions from the user's text and voice in real time.

[0418] Input: User's text data, voice data

[0419] Output: Recognized emotion data

[0420] Specific operation: Calls an emotion engine (e.g., IBM Watson's emotion analysis API) and analyzes emotions.

[0421] Step 6:

[0422] The device adjusts the tone and dialogue of the avatar based on information from the emotion engine.

[0423] Input: Recognized emotion data

[0424] Output: Adjusted dialogue

[0425] Specific behavior: Based on the emotion data, the avatar generates dialogue content and responds to the user. For example, if the user is happy, the avatar responds in a cheerful tone, saying, "That's great! What happened?"

[0426] (Application example 1)

[0427] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0428] Conventional content distribution services have had the problem of not being able to adequately provide personalized content recommendations based on the user's emotional state or individual hobbies and interests. It was also difficult to grasp the user's emotional state in real time and provide appropriate content accordingly.

[0429] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., a means for AI to reproduce the user's memories and personality using the data as training data, a means for generating an avatar of the user on the cloud, a means including an emotion engine that grasps the user's emotional state in real time, and a means for recommending optimal content according to the user's emotional state. This enables personalized content recommendations according to the user's emotional state and individual hobbies and interests.

[0430] A "messenger app" is an application that allows users to send and receive text messages, voice messages, images, videos, etc.

[0431] An "Internet search tool" is software or a web service that allows users to search for information on the Internet.

[0432] "Life log data" is data recorded about a user's daily life behavior and activities.

[0433] "Feature data" is data that represents individual characteristics of a user, such as their writing or voiceprint.

[0434] "Training data" is a dataset used to train a machine learning model.

[0435] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence.

[0436] "Means of recreating the user's memories and personality" refers to technology that recreates a user's past behavior and personality based on collected data.

[0437] The "cloud" is a collection of computer resources and services provided over the Internet.

[0438] An "avatar" is a digital avatar that acts as a user's alter ego.

[0439] The "emotion engine" is a technology that recognizes the user's emotional state in real time and generates a corresponding response.

[0440] "Content recommendation" is a technology that suggests optimal content based on a user's hobbies, interests, and emotional state.

[0441] A system for implementing this invention is configured as follows: First, life log data and characteristic data such as text and voiceprints are collected from messenger apps and internet search tools installed on a user's smartphone or personal computer. This data reflects the user's behavioral patterns, hobbies, interests, personality, etc.

[0442] The server uses the collected life log data and feature data as training data and uses a generative AI model to recreate the user's memories and personality. This generative AI model is implemented using a programming language such as Python and runs on the cloud. A digital avatar of the user is generated on the cloud, enabling interactions that reflect the user's memories and personality.

[0443] In addition, the app is equipped with an emotion engine that understands the user's emotional state in real time. This emotion engine analyzes the user's text and voice messages to recognize their emotional state. For example, if a user types "I'm a little tired today," the emotion engine will detect the user's sense of fatigue.

[0444] The server then recommends optimal content to the user based on the emotional state obtained from the emotion engine. Content recommendations are based on the user's behavioral patterns and emotional state, and suggest movies, music, and other content that will help the user relax.

[0445] For example, if a user types "I feel a bit tired today," the emotion engine will detect the user's sense of fatigue and recommend relaxing movies and music. This process is achieved by inputting the following prompt sentence into the generative AI model:

[0446] "Analyze the user's emotional state and recommend relaxing content. The user's input is 'I'm a little tired today.'"

[0447] In this way, personalized content recommendations based on the user's emotional state and individual hobbies and interests become possible.

[0448] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0449] Step 1:

[0450] Users use messenger apps and internet search tools installed on their smartphones or personal computers to send messages and perform searches on a daily basis. This generates lifelog data and feature data such as sentences and voiceprints. The input is the user's messages and search queries, and the output is the lifelog data and feature data.

[0451] Step 2:

[0452] The device transmits the generated life log data and feature data to the server. The input is the life log data and feature data, and the output is the data transmitted to the server. Specifically, the device performs data collection and transmission.

[0453] Step 3:

[0454] The server inputs the received life log data and feature data into a generative AI model as training data to recreate the user's memories and personality. The input is life log data and feature data, and the output is a model that recreates the user's memories and personality. Specific operations include data preprocessing, model training, and model generation.

[0455] Step 4:

[0456] The server generates a digital avatar of the user on the cloud. The input is a model that recreates the user's memories and personality, and the output is a digital avatar on the cloud. Specifically, the avatar is generated and placed on the cloud.

[0457] Step 5:

[0458] When a user inputs a message, the terminal sends the message to the server. The input is the user's message, and the output is the message sent to the server. The specific operation is to send the message.

[0459] Step 6:

[0460] The server inputs the received message into the emotion engine and grasps the user's emotional state in real time. The input is the user's message, and the output is the user's emotional state. Specifically, the engine analyzes the message and recognizes the user's emotional state.

[0461] Step 7:

[0462] The server recommends optimal content based on the emotional state obtained from the emotion engine. The input is the user's emotional state, and the output is the recommended content. Specifically, the server evaluates the emotional state and selects content.

[0463] Step 8:

[0464] The server sends the recommended content to the terminal and presents it to the user. The input is the recommended content, and the output is the content presented to the user. Specifically, the content is sent and displayed.

[0465] In this way, personalized content recommendations based on the user's emotional state and individual hobbies and interests are realized.

[0466] Example 2

[0467] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0468] In modern society, users are often so busy that they neglect communication with acquaintances, friends, and family. Furthermore, even after a user passes away, there is a need for those left behind to continue to communicate with them. However, with conventional technology, it has been difficult to accurately reproduce a user's language and thought patterns, and to engage in appropriate conversations based on their emotional state.

[0469] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0470] In this invention, the server includes means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., means for artificial intelligence to use this data as training data to reproduce the user's memories and personality, means for generating an avatar of the user on the cloud, means for analyzing the user's emotional state, means for generating dialogue for the avatar based on the results of the emotion analysis, and means for executing the generated dialogue. This makes it possible for the user to maintain communication with acquaintances, friends, and family even when they are absent or busy, or even after they have passed away.

[0471] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[0472] An "Internet search tool" is software that allows users to search for information on the Internet and obtain the data they need.

[0473] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, health data, etc.

[0474] "Characteristic data" is data that represents a user's individual characteristics, including writing style and voiceprint.

[0475] "Artificial intelligence" is the technology that enables computers to mimic human intelligence and learn, reason, and self-correct.

[0476] The "cloud" is a collection of computer resources and services provided over the Internet.

[0477] An "avatar" is a virtual entity that mimics the user's speech and thought patterns and converses on the user's behalf.

[0478] An "emotional state" is a state that indicates the user's current emotion, and includes joy, anger, sadness, etc.

[0479] "Sentiment analysis" is the process of reading and analyzing emotions from a user's text data or voice data.

[0480] "Dialogue generation" is the process of generating appropriate responses based on the user's emotional state and past dialogue history.

[0481] "Dialogue execution" is the process of actually sending the generated dialogue on behalf of the user.

[0482] This invention is a system that generates a user's avatar on the cloud and allows the user to maintain communication with acquaintances, friends, and family even when the user is absent, busy, or has passed away. Specific embodiments of this system are described below.

[0483] First, the server collects user data. Specifically, it acquires life log data and characteristic data such as sentences and voiceprints from messenger apps and internet search tools. This data is used as training data to learn the user's vocabulary and thought patterns.

[0484] The server then analyzes the collected data and learns the user's language and thought patterns. Specifically, it analyzes the data using natural language processing technology and uses a generative AI model (such as Google Cloud Natural Language API or OpenAI's GPT-4) to learn the user's characteristics.

[0485] The server then generates an avatar of the user based on the learned model. This avatar mimics the user's language and thought patterns and interacts on the user's behalf. The avatar is placed in the cloud and sends and receives messages on the user's behalf.

[0486] Furthermore, the server uses an emotion engine to analyze the user's emotional state. Specifically, it analyzes the user's text data and voice data to grasp the user's emotional state in real time. For example, it performs emotion analysis using Microsoft® Azure®'s Text Analytics for Sentiment Analysis.

[0487] Based on the emotion analysis results, the server generates dialogue for the avatar. For example, if the user is angry, it will choose words of apology, and if the user is happy, it will choose words of congratulations. The generated dialogue will be appropriate depending on the user's emotional state.

[0488] Finally, the device executes the generated dialogue and sends messages on behalf of the user, allowing users to maintain communication with acquaintances, friends, and family even when they are away or busy.

[0489] As a specific example, if a user is too busy to reply, the avatar will send a message on their behalf. For example, if the user receives a message from a friend asking, "What are your plans for this weekend?", the avatar will reply, "I'm planning to spend this weekend with my family," based on the user's past conversation history. Also, if the user is angry, the emotion engine will detect that emotion and the avatar will choose words of apology. For example, if the user sends an angry message saying, "Why did this happen?", the avatar will reply, "I'm sorry, I'll deal with it right away."

[0490] An example of a prompt sentence might be:

[0491] "Based on the user's past messaging history, generate an appropriate reply to the following message: 'What are your plans this weekend?'"

[0492] "Generate an appropriate response for when a user is angry. User message: 'Why did this happen to me?'"

[0493] In this way, the server and the terminal work together to generate an avatar of the user and use the emotion engine to carry out appropriate dialogue according to the user's emotional state.

[0494] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0495] Step 1:

[0496] The server collects user data. Specifically, it acquires life log data and feature data such as sentences and voiceprints from messenger apps and internet search tools. User account information and access permissions are required as input, and the collected data is obtained as output. For example, email data is collected using the Gmail API.

[0497] Step 2:

[0498] The server analyzes the collected data and learns the user's language usage and thought patterns. Specifically, it uses natural language processing technology to analyze the data and uses a generative AI model (for example, Google Cloud Natural Language API or OpenAI's GPT-4). The collected data is required as input, and a model that captures the user's characteristics is generated as output. For example, past message history can be analyzed to extract the user's language usage patterns.

[0499] Step 3:

[0500] The server generates an avatar for the user based on the trained model. The avatar imitates the user's language and thought patterns and converses on behalf of the user. The trained model is required as input, and an avatar deployed on the cloud is generated as output. For example, an avatar for the user is created and deployed on the cloud.

[0501] Step 4:

[0502] The server uses an emotion engine to analyze the user's emotional state. Specifically, it analyzes the user's text data and voice data to grasp their emotional state in real time. The input requires the user's latest messages and voice data, and the output is the emotion analysis results. For example, emotion analysis can be performed using Microsoft Azure's Text Analytics for Sentiment Analysis.

[0503] Step 5:

[0504] The server generates the avatar's dialogue based on the emotion analysis results. For example, if the user is angry, it chooses words of apology, and if the user is happy, it chooses words of congratulations. The input requires the emotion analysis results and a model that captures the user's characteristics, and the output is the generated dialogue. For example, if the user sends an angry message such as "Why did this happen?", the avatar will reply, "I'm sorry, I'll deal with it right away."

[0505] Step 6:

[0506] The device executes the generated dialogue and sends messages on behalf of the user. It takes the generated dialogue as input and the sent message as output. For example, if the user's friend sends a message asking "What are your plans this weekend?", the avatar replies "I'm planning to spend this weekend with my family."

[0507] In this way, the server and the terminal work together to generate an avatar of the user and use the emotion engine to carry out appropriate dialogue according to the user's emotional state.

[0508] (Application example 2)

[0509] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0510] In modern society, users are often too busy to fully utilize content distribution services, and there are also issues with not being able to communicate appropriately when users are away. There is also the issue of it being difficult to respond appropriately to users' emotional states. Furthermore, there is a lack of ways to maintain communication with acquaintances, friends, and family members even after a user passes away.

[0511] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0512] In this invention, the server includes a means for linking life log data from messenger apps, internet search tools, etc. with characteristic data such as text and voiceprints; a means for AI to use this data as training data to recreate the user's memories and personality; a means for generating an avatar of the user on the cloud; a means for recommending content based on the user's viewing history and ratings; a means for posting comments that imitate the user's language and thought patterns; and a means for controlling the avatar's behavior according to the user's emotional state. This allows users to fully utilize content distribution services even when they are busy, enables appropriate communication even when the user is absent, and enables appropriate responses according to the user's emotional state. Furthermore, it allows users to maintain communication with acquaintances, friends, and family even after their death.

[0513] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[0514] An "Internet search tool" is software that allows users to search for information on the Internet.

[0515] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, health data, etc.

[0516] "Characteristic data" is data used to identify an individual, such as a user's writing or voiceprint.

[0517] "Teacher data" is data used to train a machine learning model, and consists of pairs of input data and corresponding correct answer data.

[0518] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[0519] "Means of recreating a user's memories and personality" refers to technology that imitates a user's thought patterns and language usage based on the user's past actions and statements.

[0520] "Means for generating a person's avatar on the cloud" refers to a technology that creates a digital avatar of a user on a server accessible via the Internet.

[0521] "Viewing history" is a record of content that a user has viewed in the past.

[0522] "Ratings" are records of ratings and feedback given by users to content they have viewed.

[0523] "Means for recommending content" refers to technology that suggests new content that may interest users based on their viewing history and ratings.

[0524] "Method of posting comments that imitate language and thought patterns" refers to a technology that generates and posts comments that a user is likely to make based on the user's past comments.

[0525] "Means for controlling the behavior of the avatar according to the emotional state" refers to technology that analyzes the user's emotions and adjusts the avatar's behavior and speech based on the results.

[0526] An embodiment of the present invention will be described. First, the entire system is built on the cloud and operates in conjunction with the user's device (smartphone or head-mounted display). The system is composed of the following main components:

[0527] 1. Data Collection

[0528] The server collects life log data and feature data (such as text and voiceprints) from messenger apps and internet search tools. This data includes the user's behavioral history and what they say.

[0529] 2. Incarnation generation

[0530] The server uses the collected data as training data to train a generative AI model (e.g., GPT-4). This model is used to recreate the user's memories and personality, specifically, mimicking the user's speech and thought patterns.

[0531] 3. Cloud Avatar

[0532] The server creates an avatar of the user on the cloud. This avatar communicates on the user's behalf when the user is absent or busy. It also maintains communication with acquaintances, friends, and family even after the user's death.

[0533] 4. Content Recommendation

[0534] The server uses a generative AI model to recommend new content based on the user's viewing history and ratings, for example, suggesting the next movie to watch based on the genre of movies the user has recently watched.

[0535] Prompt Sentence Examples

[0536] "The genre of movies the user recently watched was action. Please recommend three movies that would be good next."

[0537] 5. Post a comment

[0538] The server mimics the user's language and thought patterns and posts comments on their behalf when they are absent or busy, allowing them to maintain their presence.

[0539] Prompt Sentence Examples

[0540] "Generate comments that users post after watching 'Inception'."

[0541] 6. Emotional Response

[0542] The server uses an emotion engine (e.g., Emotion API) to analyze the user's emotional state. Based on the analysis results, the avatar's behavior and speech are adjusted. For example, if the user is sad, it posts encouraging words, and if the user is happy, it posts congratulatory words.

[0543] In this way, users can fully utilize content distribution services even when they are busy, and appropriate communication can be maintained even when the user is absent. It also enables appropriate responses based on the user's emotional state, and allows communication with acquaintances, friends, and family to continue even after the user's death.

[0544] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0545] Step 1:

[0546] The server collects life log data and feature data (such as text and voiceprints) from messenger apps and internet search tools. The input is the user's behavioral history and comments, and the output is to store this data on the cloud. Specifically, the data is obtained through an API and stored in an AWS (registered trademark) S3 bucket.

[0547] Step 2:

[0548] The server uses the collected data as training data to train a generative AI model (e.g., GPT-4). The input is the collected life log data and feature data, and the output is a model for reproducing the user's memories and personality. Specifically, the data is preprocessed and fed into the AI ​​model for learning.

[0549] Step 3:

[0550] The server generates a user avatar on the cloud. The input is a trained AI model, and the output is a digital avatar of the user. Specifically, it deploys the AI ​​model to the server on the cloud and provides an API to generate the user avatar.

[0551] Step 4:

[0552] The server uses a generative AI model to recommend new content based on the user's viewing history and ratings. The input is the user's viewing history and ratings, and the output is the recommended content. Specifically, the server analyzes the viewing history and ratings and inputs prompts into the AI ​​model to generate recommendation results.

[0553] Step 5:

[0554] The server imitates the user's language and thought patterns and posts comments on their behalf when they are absent or busy. The input is the user's past comment data, and the output is the generated comment. Specifically, it analyzes the past comment data and inputs prompt sentences into the AI ​​model to generate comments.

[0555] Step 6:

[0556] The server uses an emotion engine (e.g., Emotion API) to analyze the user's emotional state. The input is the user's current emotional data, and the output is the analysis result. Specifically, the server acquires emotional data in real time and analyzes it using the Emotion API.

[0557] Step 7:

[0558] The server adjusts the avatar's behavior and speech based on the analysis results. The input is the emotion engine's analysis results, and the output is the adjusted avatar's behavior and speech. Specifically, it inputs prompt statements into the AI ​​model based on the analysis results, generating appropriate behavior and speech.

[0559] Example 3

[0560] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0561] Previous systems had low accuracy in reproducing the user's memories and personality, and had difficulty responding in real time to the user's emotional state. Furthermore, they lacked long-term data collection and analysis, making it impossible to accurately reflect the user's emotional state. This limited the user experience and reduced the system's usefulness.

[0562] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a means for linking life log data and feature data such as text and voiceprints from a messenger app, an internet search tool, etc., a means for an AI to reproduce the user's memory and personality using the data as training data, a means for generating an avatar of the user on the cloud, a means for detecting the user's emotional state in real time, a means for controlling the avatar's facial expression according to the user's emotional state, and a means for increasing the degree of reproduction the longer the monthly subscription fee is paid. This makes it possible to reproduce the user's memory and personality with high accuracy and to react in real time according to the emotional state.

[0563] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc. in real time.

[0564] An "Internet search tool" is software that allows users to search for information on the Internet and obtain the data they need.

[0565] "Life log data" refers to data that includes behavioral history and activity records related to a user's daily life.

[0566] "Characteristic data" is unique data used to identify an individual, such as a user's writing or voiceprint.

[0567] "Teacher data" is known input-output pair data used to train a machine learning model.

[0568] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence to learn, reason, and self-correct.

[0569] The "cloud" is a collection of computer resources and services provided over the Internet.

[0570] An "avatar" is a virtual being that reproduces the user's memories and personality and interacts and acts on the user's behalf.

[0571] An "emotional state" is a state that indicates the user's current feelings or mood.

[0572] "Real-time" refers to the instantaneous collection, analysis, and reaction of data.

[0573] A "monthly subscription" is a pricing system in which users pay a fixed monthly fee to use a service.

[0574] "Reproducibility" refers to the degree to which the system can accurately reproduce the user's memories and personality.

[0575] MODE FOR CARRYING OUT THE INVENTION

[0576] This invention is a system that reproduces the user's memories and personality with high accuracy and responds in real time according to the user's emotional state. Specific embodiments of this system are described below.

[0577] 1. Program Generation

[0578] The server collects user data and uses a generative AI model to generate a program that recreates the user's memories and personality, including an emotion engine that controls the avatar's facial expressions according to the user's emotional state.

[0579] 2. Program processing explanation

[0580] As users continue to pay their monthly subscription, the server collects more data, including their behavioral history, emotional state, and conversational content. The collected data is analyzed using a generative AI model (e.g., OpenAI's GPT-4) and used to recreate the user's memories and personality.

[0581] The device uses hardware such as a camera and microphone to detect the user's emotional state in real time. For example, the camera captures the user's facial expressions, and the microphone analyzes the user's tone of voice. This data is sent to an emotion engine to determine the user's emotional state.

[0582] The emotion engine controls the facial expression of the avatar based on the user's emotional state. For example, when the user is happy, the emotion engine makes the avatar smile. When the user is sad, the emotion engine makes the avatar sad.

[0583] 3. Examples of concrete examples and prompts

[0584] As a concrete example, consider the case where a user has been paying for a service for a year. Over that time, the system has collected a huge amount of data about the user's behavioral history and emotional state. Based on this data, the generative AI model can reproduce the user's memories and personality with a very high degree of accuracy.

[0585] Example prompt sentence:

[0586] Generate a program to recreate the user's memories and personality based on data from a user who has continued paying for a year. Also, configure the emotion engine so that the avatar smiles when the user is happy and sad when the user is sad.

[0587] In this way, the server collects data and uses the generative AI model to generate a program that reproduces the user's memories and personality. The terminal detects the user's emotional state, and the emotion engine controls the avatar's facial expressions. As the user continues to pay over a long period of time, the system collects more data, and the avatar's reproduction becomes more accurate. The flow of the identification process in Example 3 is explained using Figure 21.

[0588] Program processing flow

[0589] Step 1: Collect user data

[0590] Users log in to the system and engage in everyday activities and interactions. The device uses a camera and microphone to capture the user's facial expressions and tone of voice in real time, and also collects text data entered by the user and their behavioral history.

[0591] Input: User's facial expression data, tone of voice, text data, behavioral history

[0592] Output: Collected user data

[0593] Specific behavior:

[0594] The device's camera captures the user's face and collects facial expression data.

[0595] The device's microphone records the user's voice and analyzes the tone and emotion of the voice.

[0596] Collect text data entered by users in chat.

[0597] Step 2: Sending data to the server

[0598] The device periodically sends the collected data to a server, where it is encrypted and securely transmitted.

[0599] Input: Collected user data

[0600] Output: Data sent to the server

[0601] Specific behavior:

[0602] The data collected by the device is encrypted and sent to the server.

[0603] The server stores the received data in a database.

[0604] Step 3: Data analysis on the server

[0605] The server analyzes the received data to identify the user's behavioral patterns and emotional state using data mining techniques and machine learning algorithms.

[0606] Input: Data sent to the server

[0607] Output: Analysis results (user behavior patterns, emotional state)

[0608] Specific behavior:

[0609] The server retrieves the data from the database and applies an analysis algorithm.

[0610] The server identifies the user's behavioral patterns and emotional state and stores the results.

[0611] Step 4: Recreate user memories and personality with a generative AI model

[0612] The server uses a generative AI model (e.g., OpenAI's GPT-4) based on the analysis results to recreate the user's memories and personality. The generated model then simulates the user's interactions and behavior.

[0613] Input: Analysis results

[0614] Output: Recreated user's memories and personality

[0615] Specific behavior:

[0616] The server inputs the analysis results into a generative AI model to recreate the user's memories and personality.

[0617] The generated model simulates user interactions and behavior.

[0618] Step 5: Detecting the user's emotional state with the emotion engine

[0619] The device uses an emotion engine to detect the user's emotional state in real time. The emotion engine analyzes data from the camera and microphone to determine the user's emotional state.

[0620] Input: Real-time data from camera and microphone

[0621] Output: User's emotional state

[0622] Specific behavior:

[0623] The device's emotion engine analyzes facial expression data from the camera to determine the user's emotional state.

[0624] The device's emotion engine analyzes the tone of voice from the microphone to determine the user's emotional state.

[0625] Step 6: Avatar Facial Expression Control

[0626] After the emotion engine determines the user's emotional state, the device controls the avatar's facial expression. For example, if the user is happy, the avatar will smile. If the user is sad, the avatar will make a sad expression.

[0627] Input: User's emotional state

[0628] Output: Avatar facial expression

[0629] Specific behavior:

[0630] The device controls the avatar's facial expressions based on the judgment results of the emotion engine.

[0631] The avatar makes appropriate facial expressions depending on the user's emotional state.

[0632] (Application example 3)

[0633] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0634] Conventional avatar systems focus on recreating the user's memories and personality, but they have the drawback of being unable to provide real-time responses or personalized content based on the user's emotional state. Furthermore, they lack the means to analyze the user's emotional state, which can lead to a decline in the quality of the user experience.

[0635] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., means for AI to reproduce the user's memories and personality using these as training data, means for generating an avatar of the user on the cloud, means for analyzing the user's emotional state in real time, and means for providing personalized content according to the analyzed emotional state. This makes it possible to provide real-time responses and personalized content according to the user's emotional state.

[0636] A "messenger app" is software that allows users to send and receive text and voice messages.

[0637] An "Internet search tool" is software for searching for information on the Internet.

[0638] "Life log data" refers to data related to a user's daily life, including behavioral history and location information.

[0639] "Characteristic data" is data that indicates the individual characteristics of a user, and includes sentences, voiceprints, and the like.

[0640] "Training data" is data used to train a machine learning model.

[0641] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence.

[0642] "Reproducing the user's memories and personality" means imitating the user's personality and thought patterns based on the user's past actions and statements.

[0643] "Creating an avatar of the user on the cloud" means creating a virtual avatar of the user on a server accessible via the Internet.

[0644] "Analyzing the user's emotional state in real time" means instantly determining the user's emotions at any given time based on their facial expressions, voice, etc.

[0645] "Providing personalized content" means providing optimal information and entertainment based on a user's individual preferences and emotional state.

[0646] A system for implementing the present invention uses the following hardware and software to provide personalized content according to the user's emotional state.

[0647] Hardware

[0648] 1. Camera: Used to capture the user's face. This uses the camera built into the smartphone or head-mounted display.

[0649] 2. Server: Use cloud servers to process and store data.

[0650] software

[0651] 1. OpenCV: A library for face detection. It analyzes images captured by a camera and detects faces.

[0652] 2. Keras: A deep learning library for implementing emotion recognition models. It predicts user emotions from detected facial images.

[0653] 3. Requests: A library for communicating with content delivery servers. It is used to obtain content according to the user's emotional state.

[0654] Data processing and calculation

[0655] 1. Data capture: Capture video from the camera in real time and perform face detection. Detect faces using OpenCV, convert the detected faces to grayscale, and adjust their size.

[0656] 2. Emotion prediction: The adjusted facial image is input into the Keras emotion recognition model to predict the user's emotion. There are seven types of emotions: "anger," "disgust," "fear," "happiness," "sadness," "surprise," and "neutral."

[0657] 3. Content Serving: Based on the predicted emotion, the Requests library is used to send a request to the content delivery server to retrieve the appropriate content, which is then displayed to the user.

[0658] Specific examples

[0659] For example, when a user is using a smartphone, if the camera captures the user's face and recognizes the emotion as "happy," the content distribution server will provide "comedy movies" or "fun music playlists."

[0660] Prompt Sentence Examples

[0661] "Develop an application that analyzes user emotions in real time and provides personalized content based on those emotions. Use a Keras model for emotion recognition and the Requests library for communication with the content delivery server."

[0662] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0663] Step 1:

[0664] The device uses a camera to capture the user's face in real time. The input is video data from the camera, and the output is the captured image frame. Specifically, the device starts the camera and continuously captures video.

[0665] Step 2:

[0666] The device uses OpenCV to detect faces in captured image frames. The input is the captured image frame, and the output is the detected face area. Specifically, the device converts the image to grayscale and applies a face detection algorithm to identify the face location.

[0667] Step 3:

[0668] The device converts the detected face area to grayscale and resizes it to 48x48 pixels. The input is the detected face area, and the output is the resized face image. Specifically, the device normalizes the face image and converts it into a format suitable for machine learning models.

[0669] Step 4:

[0670] The device predicts emotions from resized facial images using a Keras emotion recognition model. The input is the resized facial image, and the output is the predicted emotion label. Specifically, the device inputs the facial image into the model, calculates the probability distribution of emotions, and selects the emotion with the highest probability.

[0671] Step 5:

[0672] The device sends a request to the content delivery server using the Requests library based on the predicted emotion label. The input is the predicted emotion label, and the output is the content obtained from the content delivery server. Specifically, the device sends the emotion label in JSON format to the server and requests the appropriate content.

[0673] Step 6:

[0674] The server selects personalized content based on the received emotion label and sends it back to the device. The input is the emotion label and the output is the selected content. Specifically, the server searches the database for content corresponding to the emotion label and sends it to the device.

[0675] Step 7:

[0676] The terminal displays the content received from the server to the user. The input is the content received from the server, and the output is the content displayed to the user. In concrete terms, the terminal displays the received content on the screen so that the user can view or use it.

[0677] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0678] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0679] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0680] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0681] [Second embodiment]

[0682] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0683] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0684] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0685] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0686] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0687] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0688] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0689] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0690] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0691] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0692] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0693] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0694] "Example 1"

[0695] In one embodiment of the present invention, life log data and characteristic data such as text and voiceprints are collected from the user's daily use of messenger apps, internet search tools, etc. This data reflects the user's behavioral patterns, hobbies, interests, personality, etc. By using this data as training data and training AI, the user's memories and personality are reproduced.

[0696] "Example 2"

[0697] Next, an avatar of the user is generated on the cloud. This avatar is designed to maintain communication with acquaintances, friends, and family when the user is absent, busy, or even after the user has passed away. Specifically, the avatar mimics the user's speech and thought patterns, recreating how the user normally interacts.

[0698] "Example 3"

[0699] Furthermore, the system of the present invention employs a monthly subscription system. The longer a user continues to pay, the more data the system collects and the more the AI ​​learns. This allows the avatar to be more accurately recreated, and the user's memories and personality to be more accurately reproduced.

[0700] The processing flow of each embodiment will be described below.

[0701] "Example 1"

[0702] Step 1: Collect life log data and characteristic data such as text and voiceprints from messenger apps, internet search tools, etc. that users use on a daily basis.

[0703] Step 2: The collected data is used as training data for the AI ​​to learn. Through this learning, the AI ​​will understand the user's behavioral patterns, hobbies, interests, personality, etc.

[0704] Step 3: As the AI ​​learns, it becomes better at replicating the user's memories and personality.

[0705] "Example 2"

[0706] Step 1: Create an avatar of the user on the cloud. This avatar will maintain communication with acquaintances, friends, and family when the user is absent, busy, or even after the user has passed away.

[0707] Step 2: The avatar mimics the user's language and thought patterns, replicating how the user normally interacts.

[0708] "Example 3"

[0709] Step 1: The system of the present invention employs a monthly billing system.

[0710] Step 2: The longer the user continues to pay, the more data the system collects and the more the AI ​​learns.

[0711] Step 3: This increases the fidelity of the avatar, allowing it to more accurately recreate the user's memories and personality.

[0712] Example 1

[0713] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0714] Modern society demands technology that can collect data reflecting a user's behavioral patterns, interests, and personality, and then use that data to recreate their memories and personality. However, existing systems lack integration between the steps of data collection, preprocessing, storage, learning, evaluation, and generation, making it difficult to efficiently recreate a user's memories and personality. Furthermore, there is a lack of ways for users to maintain communication with acquaintances, friends, and family, even when they are absent or have passed away. Furthermore, there is no established sustainable billing model for improving the level of recreation.

[0715] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0716] In this invention, the server includes: means for collecting life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc.; means for preprocessing the collected data and converting it into a format suitable for learning an AI model; means for saving the preprocessed data in a database; means for inputting the saved data as training data into an AI model and training it to reproduce the user's memories and personality; means for evaluating the AI ​​model after training and confirming its performance; means for saving well-rated models; and means for generating an avatar of the user on the cloud. This allows for efficient collection and processing of data reflecting the user's behavioral patterns, interests, and personality, enabling highly accurate reproduction of the user's memories and personality. Furthermore, the system allows the user to maintain communication with acquaintances, friends, and family even when they are absent or after their death, and the level of reproduction can be improved through a sustainable billing model.

[0717] A "messenger app" is a software application that enables users to send and receive text messages, voice messages, images, videos, etc.

[0718] An "Internet search tool" is a software application that allows a user to search for information on the Internet.

[0719] "Life log data" is data that records a user's actions and activities in their daily life.

[0720] "Characteristic data" is data that reflects personal characteristics, such as a user's writing style or voiceprint.

[0721] "Preprocessing" is the process of converting collected data into a format suitable for learning AI models.

[0722] A "database" is a system for efficiently storing, managing, and retrieving data.

[0723] "Training data" is data with correct labels that is used to train an AI model.

[0724] An "AI model" is a mathematical model that uses artificial intelligence technology to analyze data and make predictions and classifications.

[0725] "Learning" is the process by which an AI model discovers patterns and rules based on training data.

[0726] "Evaluation" is the process of checking the performance of an AI model after it has completed training.

[0727] The "cloud" is a collection of computer resources and services provided over the Internet.

[0728] An "avatar" is a digital being that recreates the user's memories and personality.

[0729] A "charging model" is a business model for collecting fees for the use of a service.

[0730] MODE FOR CARRYING OUT THE INVENTION

[0731] This invention is a system that collects life log data and characteristic data such as sentences and voiceprints from messenger apps and internet search tools that users use on a daily basis, and recreates the user's memories and personality based on this data. Specific embodiments of this system are described below.

[0732] Data collection

[0733] The server collects life log data and characteristic data such as sentences and voiceprints from the messenger apps and internet search tools that users use on a daily basis. This data is collected using common software such as messenger apps and internet search tools. The collected data reflects the user's behavioral patterns, hobbies, interests, personality, etc.

[0734] Data Preprocessing

[0735] The server preprocesses the collected data and converts it into a format suitable for learning AI models. Specifically, it tokenizes text data and converts audio data into a spectrogram. This is done using Python libraries such as NLTK and Librosa. For example, NLTK is used to tokenize text data, and Librosa is used to convert audio data into a spectrogram.

[0736] Data storage

[0737] The server stores the preprocessed data in a database, for example using a database system such as MySQL or MongoDB, by establishing a database connection and inserting the preprocessed data into the appropriate tables and collections.

[0738] Training an AI model

[0739] The server inputs the preprocessed data into the AI ​​model as training data, and trains it to reproduce the user's memories and personality. This training uses generative AI models such as GPT-4 and BERT. Specifically, it initializes the model, batches the data, and executes the training.

[0740] Evaluating and saving the model

[0741] The server evaluates the AI ​​model after training and checks its performance. Evaluation uses metrics such as precision and recall. Models that pass the evaluation are stored in a file system or cloud storage.

[0742] Avatar generation on the cloud

[0743] The server uses the well-reviewed model to generate a cloud-based avatar of the user, which recreates the user's memories and personality and is used to maintain communication with acquaintances, friends, and family.

[0744] Specific examples

[0745] Example 1: Data collection from messenger apps

[0746] The server collects messages sent and received by users on popular messaging apps. These messages reflect the user's conversation patterns and interests. This message data is preprocessed and trained into an AI model to reproduce the user's conversation style.

[0747] Example 2: Data collection from internet search tools

[0748] The server collects search queries performed by users using common internet search tools. The collected search queries reflect the user's interests and concerns. This search data is preprocessed and trained into an AI model to reproduce the user's interests.

[0749] Prompt Sentence Examples

[0750] "Create a program to collect messages sent by users on popular messaging apps and train an AI model to replicate the user's conversational style."

[0751] "You will create a program that collects search queries performed by users on popular internet search tools and trains an AI model to replicate the user's interests."

[0752] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0753] Step 1: Data collection

[0754] The server collects life log data and feature data such as sentences and voiceprints from the messenger apps and internet search tools that users use on a daily basis. It receives the log data from the messenger apps and internet search tools as input and generates the collected raw data as output. Specifically, the server obtains data using APIs and scraping technology, and collects the data using secure communication protocols (e.g., HTTPS).

[0755] Step 2: Data Preprocessing

[0756] The server preprocesses the collected data and converts it into a format suitable for training the AI ​​model. It receives the collected raw data as input and produces preprocessed data as output. Specifically, the server performs the following operations:

[0757] Tokenizing text data: Splitting sentences into words and phrases using Python's NLTK library.

[0758] Spectrogram conversion of audio data: Use the Librosa library in Python to convert audio data into a frequency spectrum.

[0759] Step 3: Save your data

[0760] The server stores the preprocessed data in a database. It receives the preprocessed data as input and generates the data stored in the database as output. Specifically, the server performs the following operations:

[0761] Establishing a database connection: Connect to MySQL or MongoDB.

[0762] Inserting Data: Inserting the pre-processed data into the appropriate tables and collections.

[0763] Step 4: Training the AI ​​model

[0764] The server inputs the preprocessed data as training data into the AI ​​model, which then learns to replicate the user's memories and personality. As input, it receives the preprocessed data stored in the database, and as output, it generates a trained AI model. Specifically, the server performs the following operations:

[0765] Model initialization: Initialize a generative AI model such as GPT-4 or BERT.

[0766] Data Batching: Preprocessed data is fed into the model in batches.

[0767] Execute training: Input data into the model and execute training.

[0768] Step 5: Evaluate and save the model

[0769] The server evaluates the AI ​​model after training and checks its performance. As input, it receives the trained AI model and evaluation data, and as output, it generates the evaluation results and a saved model. Specifically, the server performs the following operations:

[0770] Model evaluation: Calculating metrics such as precision and recall.

[0771] Save the model: Save the model with a good evaluation to the file system or cloud storage.

[0772] Step 6: Cloud avatar generation

[0773] The server generates an avatar of the person on the cloud using the model with a good evaluation. It receives the stored AI model as input and generates the avatar on the cloud as output. Specifically, the server performs the following operations:

[0774] Deploying the model: Deploying the AI ​​model to a cloud environment.

[0775] Avatar Generation: Using the deployed model, an avatar is generated that recreates the user's memories and personality.

[0776] (Application example 1)

[0777] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0778] In modern society, there is a demand for providing optimal content for each individual user by utilizing their life log data and characteristic data. However, conventional systems have had difficulty recreating a user's memories and personality and recommending personalized content. Furthermore, there has been a lack of ways for users to maintain communication with acquaintances and family when they are absent, busy, or even after death. A new system is needed to solve these issues.

[0779] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0780] In this invention, the server includes a means for linking life log data from messenger apps, internet search tools, etc. with feature data such as text and voiceprints; a means for AI to recreate the user's memories and personality using the data as training data; a means for generating an avatar of the user on the cloud; and a means for automatically recommending content based on the user's preferences and interests based on the user's life log data and feature data. This makes it possible to recreate the user's memories and personality and provide personalized content. It also allows the user to maintain communication with acquaintances and family even when they are away, busy, or even after death.

[0781] A "messenger app" is an application that allows users to send and receive text messages, voice messages, images, videos, etc.

[0782] An "Internet search tool" is software or a web service that allows users to search for information on the Internet.

[0783] "Life log data" is data recorded about a user's daily life behavior and activities.

[0784] "Characteristic data" is data used to identify individual users, such as their writing or voiceprint.

[0785] "Training data" is a dataset used for AI learning.

[0786] "AI" stands for artificial intelligence, a technology that allows machines to mimic human intelligence.

[0787] "Reproducing the user's memories and personality" means that the AI ​​imitates the user's thinking and behavior patterns based on the user's past actions and statements.

[0788] "Creating a digital avatar on the cloud" means using cloud computing technology to create a digital avatar of a user.

[0789] "Automatic content recommendation" means that AI suggests appropriate information and entertainment based on a user's preferences and interests.

[0790] The system for implementing this invention collects user life log data and characteristic data, recreates the user's memories and personality based on the collected data, and recommends personalized content. Specific embodiments of this system are described below.

[0791] System Configuration

[0792] Hardware

[0793] Server: A server for collecting, processing, and storing data, as well as training and inferencing AI models.

[0794] Device: The device used by the user, such as a smartphone or computer.

[0795] software

[0796] Messenger app: An application that allows users to send and receive text messages, voice messages, images, videos, etc.

[0797] Internet Search Tools: Software or web services that allow users to search for information on the Internet.

[0798] AI model: A generative AI model that recreates a user's memories and personality based on their life log data and feature data, and recommends personalized content.

[0799] Data collection and processing

[0800] The server collects user life log data and feature data from messaging apps and internet search tools, including user message history, search history, and voice data. The collected data is preprocessed and then used as training data for the AI ​​model.

[0801] AI model training and inference

[0802] The server trains the AI ​​model based on the collected data. Specifically, text data is digitized using TfidfVectorizer and a clustering algorithm (e.g., KMeans) is used to create a user profile. Voice data is converted to text using speech recognition technology and processed in the same way.

[0803] Content Recommendations

[0804] The server uses a trained AI model to automatically recommend content based on the user's preferences and interests, including entertainment content such as movies, music, and articles. The recommended content is then sent to the user's device.

[0805] Specific examples

[0806] For example, if User A frequently talks about "movies" on a messaging app, the server collects that data and trains the AI ​​model. If User A's search history frequently includes searches for "latest movie reviews" and "movie trailers," the server creates a profile for User A based on this data. As a result, User A is recommended the latest movie review articles and movie trailer videos.

[0807] Prompt Sentence Examples

[0808] Develop an application that recommends content based on the user's interests and preferences, based on data collected from the user's messaging app and search history. Specifically, implement a feature that analyzes the topics and keywords frequently discussed by the user and automatically recommends content such as movies, music, and articles.

[0809] In this way, a system can be constructed that utilizes users' life log data and characteristic data to deliver personalized content.

[0810] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0811] Step 1:

[0812] The server collects user life log data and feature data from messenger apps and internet search tools, specifically, user message history, search history, voice data, etc. This data is collected using APIs and database queries.

[0813] Input: Raw data from messaging apps and search tools

[0814] Output: Collected lifelog data and feature data

[0815] Step 2:

[0816] The server preprocesses the collected data: text data is digitized using TfidfVectorizer, and voice data is converted to text using speech recognition technology, converting the data into a format suitable for the AI ​​model.

[0817] Input: Collected lifelog data and feature data

[0818] Output: Preprocessed data

[0819] Step 3:

[0820] The server trains the AI ​​model based on the preprocessed data. Specifically, it creates a user profile using a clustering algorithm (e.g., KMeans), which allows the model to reflect the user's behavioral patterns and interests.

[0821] Input: Preprocessed data

[0822] Output: A trained AI model

[0823] Step 4:

[0824] The server uses a trained AI model to automatically recommend content based on the user's preferences and interests, specifically selecting entertainment content such as movies, music, and articles based on the user profile.

[0825] Input: Trained AI model, user profile

[0826] Output: Recommended content

[0827] Step 5:

[0828] The server notifies the user of the recommended content via push notifications or in-app messages.

[0829] Input: Recommended content

[0830] Output: Content displayed on the user's device

[0831] In this way, a system is constructed that utilizes users' life log data and characteristic data to deliver personalized content.

[0832] Example 2

[0833] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0834] In modern society, there is a need to maintain communication with acquaintances, friends, and family even when users are away, busy, or have passed away. However, conventional technologies have struggled to accurately mimic a user's language and thought patterns to achieve natural dialogue. Furthermore, there has been a lack of means to effectively utilize a user's past messages and conversation history to train a generative AI model. This has led to the problem of user avatars generating unnatural responses.

[0835] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0836] In this invention, the server includes: means for linking life log data and feature data such as sentences and voiceprints from messenger apps, internet search tools, etc., means for a generative AI model to reproduce the user's memories and personality using the linked data as training data; means for generating an avatar of the user on the cloud; means for collecting the user's past messages and dialogue history and storing them in a database; means for preprocessing the collected data, cleaning the text, tokenizing, and extracting important phrases and patterns; means for training the generative AI model using the preprocessed data; means for generating an avatar of the user using the trained generative AI model; means for receiving messages from acquaintances and family members, inputting them as prompts into the generative AI model, and generating appropriate replies; and means for sending the generated replies to acquaintances and family members. This enables the system to accurately mimic the user's language and thought patterns and realize natural conversations.

[0837] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[0838] An "Internet search tool" is software that allows users to search for and retrieve information on the Internet.

[0839] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, and health data.

[0840] "Characteristic data" is data that represents a user's individual characteristics, including writing style and voiceprint.

[0841] A "generative AI model" is a model that uses artificial intelligence technology to imitate a user's language usage and thought patterns.

[0842] The "cloud" is a collection of computer resources and services provided over the Internet.

[0843] An "avatar" is a virtual entity that mimics the user's speech and thought patterns and converses on the user's behalf.

[0844] A "database" is a system for efficiently storing, searching, and managing data.

[0845] "Preprocessing" refers to the process of converting data into a format suitable for analysis and model training.

[0846] "Tokenization" is the process of dividing text data into smaller units such as words or phrases.

[0847] A "prompt" is text that is input to a generative AI model and contains instructions for the model to generate an appropriate response.

[0848] This invention is a system for maintaining communication with acquaintances, friends, and family even when the user is away, busy, or has passed away. This system links life log data from messenger apps, internet search tools, etc. with feature data such as text and voiceprints, and uses this data as training data to generate an AI model that reproduces the user's memories and personality. A specific embodiment of this system is described below.

[0849] 1. Collection of User Data

[0850] The server collects the user's past messages and conversation history. This includes emails, chat logs, and social media posts. The server stores this data in a database. For example, the server accesses the user's email account and downloads past emails. It also uses the chat app's API to retrieve the user's chat log.

[0851] 2. Data Preprocessing

[0852] The server pre-processes the collected data, which includes cleaning the text, tokenizing it, and extracting important phrases and patterns. Specifically, the server removes unnecessary HTML tags and special characters from the text, splits the text into words and phrases (tokenization), and extracts frequently occurring phrases and patterns, which are then stored in a database.

[0853] 3. Training the generative AI model

[0854] The server uses the preprocessed data to train a generative AI model. For example, OpenAI's GPT-4 is used here. The server inputs the preprocessed data into the generative AI model and trains the model. Once trained, the model will be able to mimic the user's language and thought patterns.

[0855] 4. The Creation of an Avatar

[0856] The server uses a trained generative AI model to generate an avatar for the user, which mimics the user's speech and thought patterns to replicate how the user normally interacts. The generated avatar is then stored in a database.

[0857] 5. Receiving messages and generating replies

[0858] The device receives messages from acquaintances and family members. The device inputs this message as a prompt into the generative AI model to generate an appropriate reply. For example, if a message is received from an acquaintance saying, "What are your plans for tomorrow?", the device inputs this message into the generative AI model as a prompt. An example of a prompt is "User's name: Yamada Taro, User's past message: 'I'm busy today, so please check my plans for tomorrow,' Prompt: 'When Yamada Taro is busy, please generate a reply to check tomorrow's plans.'"

[0859] 6. Sending a Reply

[0860] The device then sends the generated reply to a friend or family member. For example, it receives a reply from the generative AI model saying, "Tomorrow's schedule is a meeting at 10:00 AM and a client meeting at 2:00 PM," and sends it to a friend or family member.

[0861] In this way, the server and device work together to generate an avatar of the user, enabling them to maintain communication with acquaintances, friends, and family even when they are absent, busy, or have passed away.

[0862] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0863] Step 1: Collect user data

[0864] The server collects the user's past messages and conversation history. Input includes emails, chat logs, and social media posts. The server stores this data in a database. Specifically, the server accesses the email account and downloads past emails. It also uses the chat app's API to retrieve the user's chat log. The collected data is stored in the database as output.

[0865] Step 2: Preprocessing the data

[0866] The server preprocesses the collected data. The input includes the data collected in step 1. The server cleans the text, tokenizes it, and extracts important phrases and patterns. Specifically, the server removes unnecessary HTML tags and special characters from the text, and splits the text into words and phrases (tokenization). It then extracts frequently occurring phrases and patterns and stores them in a database. The output is the preprocessed data.

[0867] Step 3: Training the generative AI model

[0868] The server trains a generative AI model using the preprocessed data. The input includes the data preprocessed in step 2. The server inputs the preprocessed data into the generative AI model and trains the model. Specifically, the server inputs the data into the generative AI model (e.g., GPT-4) and adjusts the model parameters. The output is a fully trained generative AI model.

[0869] Step 4: Creating an Avatar

[0870] The server uses the trained generative AI model to generate an avatar for the user. The input includes the generative AI model trained in step 3. The server inputs the user's characteristics into the generative AI model to generate an avatar. Specifically, the server inputs the user's language and thought patterns into the model to generate an avatar. As an output, the generated avatar is stored in a database.

[0871] Step 5: Receiving the message and generating a reply

[0872] The device receives messages from acquaintances and family members. The input includes messages from acquaintances and family members. The device inputs these messages as prompts to the generative AI model, which then generates an appropriate reply. Specifically, the device uses the messaging app's API to obtain the received message and inputs it as a prompt to the generative AI model. The output is a reply from the generative AI model.

[0873] Step 6: Send your reply

[0874] The device sends the generated reply to the friend or family member. The input includes the reply generated in step 5. Specifically, the device uses the messaging app's API to send the generated reply. The output is the reply sent to the friend or family member.

[0875] In this way, the server and device work together to generate an avatar of the user, enabling them to maintain communication with acquaintances, friends, and family even when they are absent, busy, or have passed away.

[0876] (Application example 2)

[0877] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0878] In modern society, it is often difficult to provide customer service when users are absent or busy. Furthermore, there are limited ways to maintain communication with acquaintances, friends, and family members even after a user passes away. Furthermore, when providing customer service in a virtual store, it is necessary to mimic the user's language and thought patterns, but there is a lack of technology to achieve this.

[0879] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc.; means for AI to reproduce the user's memories and personality using the data as training data; means for generating an avatar of the user on the cloud; means for the user's avatar to interact with customers in a virtual store; means for generating answers to customer questions using a generative AI model; means for imitating the user's language and thought patterns using prompt sentences; and means for installation on a smartphone or head-mounted display. This enables customer interaction even when the user is absent or busy, and allows communication with acquaintances, friends, and family to be maintained even after the user's death. Furthermore, customer interaction in a virtual store that imitates the user's language and thought patterns is possible.

[0880] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[0881] An "Internet search tool" is software that searches for information on the Internet and provides it to users.

[0882] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, health data, etc.

[0883] "Characteristic data" is data used to identify an individual, such as a user's writing or voiceprint.

[0884] "Training data" is data used to train a machine learning model.

[0885] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[0886] The "cloud" is a collection of computer resources and services provided over the Internet.

[0887] An "avatar" is a virtual being created to mimic the user's speech and thought patterns.

[0888] A "virtual store" is a virtual store that exists on the Internet, where users can purchase goods and services.

[0889] "Customer service" refers to responding to and dealing with questions and requests from customers.

[0890] A "generative AI model" is an artificial intelligence model that generates text and speech by imitating a user's language and thought patterns.

[0891] A "prompt" is text that is input to a generative AI model, instructing the model how to respond.

[0892] A "smartphone" is a mobile phone that can connect to the Internet and run applications.

[0893] A "head-mounted display" is a display device worn on the head that provides virtual reality and augmented reality experiences.

[0894] The system for implementing this invention collects user life log data and feature data, and generates an avatar of the user on the cloud based on the collected data. A specific embodiment of this system will be described below.

[0895] First, the server collects the user's life log data and feature data from messenger apps, internet search tools, etc. Life log data includes the user's daily behavior history, location information, health data, etc. Feature data includes data that can be used to identify an individual, such as the user's writing style and voiceprint.

[0896] The server then uses this data as training data to train a generative AI model, which is then used to mimic the user's language and thought patterns. Specifically, it uses prompt sentences to mimic the user's language and thought patterns.

[0897] The server generates an avatar of the user in the cloud. This avatar exists to maintain communication with acquaintances, friends, and family even when the user is absent, busy, or even after the user passes away. The avatar can respond to customers in a virtual store. For example, if a customer asks, "What are the features of this product?", the avatar uses a generative AI model to generate an appropriate answer.

[0898] This system is implemented as an application installed on a smartphone or head-mounted display, and can improve the customer experience by allowing the avatar to respond to customers even when the user is absent or busy.

[0899] As a specific example, by using the following prompt sentence, the avatar can respond to customers by imitating the user's language and thought patterns.

[0900] Example prompt sentence:

[0901] You are a salesperson in a virtual store. Please answer the following questions by imitating the user's language and thought patterns.

[0902] Question: {question}

[0903] In this way, the avatar can respond to customers even when the user is absent or busy, and can maintain communication with acquaintances, friends, and family even after the user's death.It is also possible for the avatar to respond to customers in a virtual store by imitating the user's language and thought patterns.

[0904] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0905] Step 1:

[0906] The server collects users' life log data and feature data from messenger apps and internet search tools. The input includes the user's daily behavior history, location information, health data, text, voiceprint, etc. This data is collected and stored in a database. The output is the collected life log data and feature data.

[0907] Step 2:

[0908] The server preprocesses the collected lifelog data and feature data. The data collected in step 1 is used as input. Data processing such as data cleaning, normalization, and feature extraction is performed, and the data is converted into a format suitable for training a generative AI model. The preprocessed data is obtained as output.

[0909] Step 3:

[0910] The server uses the preprocessed data as training data to train the generative AI model. The preprocessed data from step 2 is used as input. The generative AI model analyzes the data and extracts patterns to learn the user's language and thought patterns. The output is a trained generative AI model.

[0911] Step 4:

[0912] The server generates an avatar for the user on the cloud. The generative AI model trained in step 3 is used as input. Based on the generative AI model, a virtual being is generated that mimics the user's speech and thought patterns. The output is an avatar generated on the cloud.

[0913] Step 5:

[0914] The device runs an application that allows the user's avatar to interact with customers in a virtual store. The input includes the avatar generated on the cloud and the customer's question. The device sends the customer's question to the cloud and generates an appropriate answer using a generative AI model. The output is the answer for the customer.

[0915] Step 6:

[0916] The server uses prompts to mimic the user's language and thought patterns. The input includes the customer's question and the prompt. The generative AI model mimics the user's language and thought patterns based on the prompt and generates an appropriate answer. The output is the mimicked answer.

[0917] Step 7:

[0918] The terminal displays the answer to the customer through an application installed on a smartphone or head-mounted display. It includes the answer generated in step 6 as input. The terminal displays the answer to the customer, improving the customer experience. The output is the answer displayed to the customer.

[0919] Example 3

[0920] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0921] In modern society, there is a demand for digitizing a user's memories and personality and permanently storing and reproducing them. However, conventional systems have difficulty accurately reproducing a user's memories and personality, and lack a mechanism for improving the degree of reproduction as the user uses the system over a long period of time. For this reason, there is a need for a system that can reproduce a user's memories and personality with high accuracy and further improve the accuracy with long-term use.

[0922] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0923] In this invention, the server includes means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., means for AI to reproduce the user's memories and personality using the linked data as training data, means for generating an avatar of the user on the cloud, means for collecting data provided by the user and storing it in a database, means for preprocessing the collected data and converting it into a format suitable for the generative AI model, means for training the generative AI model using the preprocessed data, means for receiving prompts from the user and generating responses using the generative AI model, and means for providing the generated responses to the user. This makes it possible to reproduce the user's memories and personality with high accuracy, and to further improve this accuracy with long-term use.

[0924] A "messenger app" is software that allows users to send and receive text and voice messages.

[0925] An "Internet search tool" is software that allows users to search for information on the Internet.

[0926] "Life log data" refers to data related to a user's daily life, including behavioral history and activity records.

[0927] "Characteristic data" is data that indicates the individual characteristics of a user, and includes sentences, voiceprints, etc.

[0928] "Training data" is data used to train a generative AI model, and is a dataset that has been assigned a correct answer label.

[0929] A "generative AI model" is an artificial intelligence model used to recreate a user's memories and personality.

[0930] The "cloud" is a collection of computer resources and services provided over the Internet.

[0931] An "avatar" is a digital being that recreates the user's memories and personality.

[0932] A "database" is a system for systematically storing and managing collected data.

[0933] "Preprocessing" refers to the process of converting collected data into a format suitable for a generative AI model.

[0934] A "prompt" is a question or request that a user enters into a system.

[0935] A "response" is the answer that a generative AI model generates in response to a prompt.

[0936] This invention is a system that digitizes a user's memories and personality and generates an avatar on the cloud. The system collects life log data and feature data from messenger apps, internet search tools, etc., and uses this data as training data for a generative AI model. A specific embodiment of this system is described below.

[0937] First, users provide life log data such as text messages, voice data, and image data through messenger apps or internet search tools. The device then sends this data to a server, which then stores the received data in a database.

[0938] The server then preprocesses the collected data. Specifically, text data is tokenized using natural language processing (NLP) techniques, audio data is converted to text using speech recognition software, and image data is subjected to feature extraction using image recognition techniques. Machine learning frameworks such as TensorFlow and PyTorch are used for this preprocessing.

[0939] The preprocessed data is used to train the generative AI model. The server uses this data to train the generative AI model, teaching it patterns to reproduce the user's memories and personality. As the learning progresses, the model's reproducibility improves.

[0940] When a user inputs a question or request into the system as a prompt, the device sends the prompt to the server, which uses a generative AI model to generate an appropriate response to the prompt, which is then sent to the device and provided to the user.

[0941] For example, if a user asks, "What is my favorite movie?", the server uses a generative AI model to search for information about "favorite movies" from the user's past data and generate a response such as, "Your favorite movie is Inception." Similarly, if a user asks, "Where was the last place I traveled?", the server generates a response such as, "You went to Hawaii last summer."

[0942] This system employs a monthly subscription system, and the longer a user continues to pay, the more data is collected and the more the AI ​​learns. This improves the reproducibility of the avatar, making it possible to more accurately reproduce the user's memories and personality. The flow of the identification process in Example 3 will be explained using Figure 15.

[0943] Step 1:

[0944] Users provide lifelog data such as text messages, voice data, and image data through messenger apps and internet search tools. The device sends this data to the server. The input is the lifelog data from the user, and the output is the data sent to the server.

[0945] Step 2:

[0946] The server stores the received data in a database. Specifically, it stores text messages, voice data, and image data in the corresponding database tables. The input is the data sent from the terminal, and the output is the data stored in the database.

[0947] Step 3:

[0948] The server preprocesses the collected data. Text data is tokenized using natural language processing (NLP) techniques to extract important keywords. Voice data is converted to text using voice recognition software. Image data is subjected to feature extraction using image recognition techniques. The input is raw data stored in a database, and the output is preprocessed data.

[0949] Step 4:

[0950] The server uses the preprocessed data to train a generative AI model. Specifically, it uses machine learning frameworks such as TensorFlow and PyTorch to train the model to learn patterns that will reproduce the user's memories and personality. The input is the preprocessed data, and the output is a trained generative AI model.

[0951] Step 5:

[0952] The user inputs a question or request to the system as a prompt. The terminal sends this prompt to the server. The input is the prompt from the user, and the output is the prompt sent to the server.

[0953] Step 6:

[0954] The server inputs the received prompt into the generative AI model to generate an appropriate response. The generative AI model generates the most appropriate answer based on the user's past data. The input is the prompt and the trained generative AI model, and the output is the generated response.

[0955] Step 7:

[0956] The server sends the generated response to the terminal, which displays this response to the user. The input is the generated response, and the output is the response provided to the user.

[0957] Specifically, when a user asks, "What's my favorite movie?", the server uses the generative AI model to search for information about "favorite movies" from the user's past data and generates a response such as, "Your favorite movie is Inception." Similarly, when a user asks, "Where was the last place I traveled?", the server generates a response such as, "You went to Hawaii last summer."

[0958] (Application example 3)

[0959] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0960] In conventional systems, the accuracy of the avatars that reproduce the user's memories and personality was low, making it difficult to recommend individually customized content. Furthermore, there was a lack of a mechanism for improving the accuracy of the system as the user uses it over a long period of time, making it difficult to improve the user experience.

[0961] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for linking life log data and feature data such as sentences and voiceprints from messenger apps, internet search tools, etc., a means for AI to reproduce the user's memories and personality using the data as training data, a means for generating an avatar of the user on the cloud, and a means for collecting user data, generating an avatar using an AI model, and recommending individually customized content. This makes it possible to reproduce the user's memories and personality with high accuracy and recommend individually customized content.

[0962] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[0963] An "Internet search tool" is software that allows users to search for information on the Internet.

[0964] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, health data, etc.

[0965] "Characteristic data" is data used to identify an individual, such as a user's writing or voiceprint.

[0966] "Training data" is data used to train a machine learning model.

[0967] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence.

[0968] "Recreating the user's memories and personality" means imitating the user's past experiences and personality based on collected data.

[0969] The "cloud" is a collection of computer resources and services provided over the Internet.

[0970] An "avatar" is a virtual being that recreates the user's memories and personality.

[0971] "Personalized Content" is information and entertainment specially selected based on a user's preferences and interests.

[0972] An "AI model" is a mathematical model that uses artificial intelligence algorithms to analyze data and make predictions and classifications.

[0973] A "monthly subscription" is a pricing system in which users can use a service by paying a fixed monthly fee.

[0974] A system for implementing this invention is configured as follows: The server has a means for linking life log data with feature data such as text and voiceprints from messenger apps, internet search tools, etc. This makes it possible to collect data about the user's daily life and data for identifying individuals.

[0975] Next, the server has a means for the AI ​​to recreate the user's memories and personality using the collected data as training data. Specifically, based on the collected life log data and feature data, the AI ​​model mimics the user's past experiences and personality. This AI model is implemented using OpenAI's API.

[0976] Furthermore, the server has a means for generating an avatar of the user on the cloud. The generated avatar is a virtual entity that reproduces the user's memories and personality, and is managed on the cloud.

[0977] The server also has a means for collecting user data, generating an avatar using an AI model, and recommending individually customized content. Specifically, the server collects data such as the user's viewing history, preferences, and feedback, and uses the AI ​​model to recommend the most suitable content to the user based on this data. This recommendation provides information and entertainment specially selected based on the user's preferences and interests.

[0978] As users continue to pay over time, the system collects more data and the AI ​​learns more, allowing the avatar to become more realistic and more accurately recreate the user's memories and personality.

[0979] As a concrete example, consider a case where a user likes "action movies" and "comedy movies," likes "movie1," and dislikes "movie2." An example of a prompt sentence that generates an avatar based on this data and recommends content is as follows:

[0980] Example prompt sentence:

[0981] Generate an avatar that recreates the user's memories and personality based on the user's data: {"user_id": "user123", "viewing_history": ["movie1", "movie2"], "preferences": ["action", "comedy"], "feedback": ["liked movie1", "disliked movie2"]}.

[0982] By inputting this prompt into OpenAI's API, an avatar that replicates the user's memories and personality is generated, and the system can then recommend the most appropriate content to the user.

[0983] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0984] Step 1:

[0985] The server collects life log data and feature data from messenger apps and internet search tools. Specifically, it obtains users' text messages, voice messages, search history, location information, etc. The input is various user data, and the output is the collected life log data and feature data.

[0986] Step 2:

[0987] The server inputs the collected life log data and feature data into the AI ​​model as training data. Specifically, it preprocesses this data and converts it into a format that the AI ​​model can learn from. The input is the preprocessed data, and the output is the training data used to train the AI ​​model.

[0988] Step 3:

[0989] The server uses an AI model to generate an avatar that replicates the user's memories and personality. Specifically, it uses OpenAI's API to generate prompts based on the user's data and input them into the AI ​​model. The input is the prompt, and the output is the generated avatar.

[0990] Step 4:

[0991] The server stores the generated avatar on the cloud. Specifically, it uses a cloud storage service to securely store the avatar data. The input is the generated avatar, and the output is the avatar data stored on the cloud.

[0992] Step 5:

[0993] The server collects data such as the user's viewing history, preferences, and feedback, and uses an AI model to recommend individually customized content. Specifically, it generates prompts based on the user's data and inputs them into the AI ​​model. The input is the user's viewing history and preferences, and the output is the recommended content.

[0994] Step 6:

[0995] The user receives the recommended content and views or uses it. Specifically, the user displays and enjoys the recommended content using a smartphone or other device. The input is the recommended content, and the output is the user's viewing or usage history.

[0996] Step 7:

[0997] As users continue to pay over time, the server collects more data and continues to train the AI ​​model. Specifically, the server periodically collects user usage data through a monthly subscription and retrains the AI ​​model. The input is continuously collected user data, and the output is the retrained AI model.

[0998] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0999] "Example 1"

[1000] One embodiment of the present invention is a system incorporating an emotion engine. This system recognizes the user's emotions and adjusts the avatar's response accordingly. Specifically, when the user is happy, the avatar will speak in a bright tone. Conversely, when the user is sad, the avatar will speak in a calm tone. In this way, the emotion engine grasps the user's emotional state in real time and adjusts the avatar's response accordingly.

[1001] "Example 2"

[1002] The emotion engine also controls the behavior of the avatar according to the user's emotional state.

[1003] When the user is angry, the avatar will choose to apologize; when the user is happy, the avatar will choose to congratulate. In this way, the emotion engine understands the user's emotional state and controls the avatar's behavior accordingly.

[1004] "Example 3"

[1005] Furthermore, the emotion engine controls the facial expression of the avatar according to the user's emotional state. For example, when the user is happy, the avatar makes a smiling face. When the user is sad, the avatar makes a sad face. In this way, the emotion engine understands the user's emotional state and controls the facial expression of the avatar accordingly.

[1006] The processing flow of each embodiment will be described below.

[1007] "Example 1"

[1008] Step 1: The emotion engine understands the user's emotional state in real time.

[1009] Step 2: The emotion engine adjusts the avatar's response according to the emotional state it has determined.

[1010] Step 3: The avatar responds with a tailored response to the user.

[1011] "Example 2"

[1012] Step 1: The emotion engine understands the user's emotional state in real time.

[1013] Step 2: The emotion engine controls the avatar's behavior according to the emotional state it has grasped.

[1014] Step 3: The avatar performs the controlled action on the user.

[1015] "Example 3"

[1016] Step 1: The emotion engine understands the user's emotional state in real time.

[1017] Step 2: The emotion engine controls the avatar's facial expression according to the emotional state it has grasped.

[1018] Step 3: The avatar displays a controlled facial expression to the user.

[1019] Example 1

[1020] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1021] In modern society, there is a demand for systems that can accurately grasp a user's behavioral patterns and emotional state and engage in dialogue accordingly. However, conventional systems have had difficulty recognizing a user's emotions in real time and generating appropriate responses accordingly. Furthermore, data collection and analysis to recreate a user's memories and personality were insufficient, making it difficult to increase user satisfaction.

[1022] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1023] In this invention, the server includes means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., means for AI to reproduce the user's memories and personality using this as training data, means for generating an avatar of the user on the cloud, means for recognizing the user's emotions in real time using an emotion engine, and means for adjusting the avatar's tone and dialogue content according to the user's emotional state. This makes it possible to have a dialogue in real time according to the user's emotional state and reproduce the user's memories and personality with high accuracy.

[1024] A "messenger app" is a software application that enables users to send and receive text messages, voice messages, images, videos, etc.

[1025] An "Internet search tool" is a software application that allows a user to search for information on the Internet.

[1026] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, and activity records.

[1027] "Text" refers to text data entered by a user, including messages, comments, posts, etc.

[1028] A "voiceprint" is characteristic data extracted from a user's voice data, and includes voice characteristics that can be used to identify an individual.

[1029] "Feature data" is data that shows specific patterns or characteristics extracted from a user's behavior or comments.

[1030] "Training data" is a dataset used to train a machine learning model, and includes input data and corresponding ground truth data.

[1031] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn, reason, recognize, and make judgments.

[1032] "Recreating the user's memories and personality" means recreating the user's past actions and statements based on collected data, and imitating the user's personality and preferences.

[1033] The "cloud" is a collection of computer resources and services provided over the Internet, and is an infrastructure for storing and processing data.

[1034] An "avatar" is a virtual entity that interacts on behalf of the user and is generated based on the user's memories and personality.

[1035] An "emotion engine" is a software component that analyzes and recognizes emotions from a user's text and voice.

[1036] "Real-time recognition" means analyzing emotions instantly in response to user input and providing results.

[1037] "Tone" refers to the tone and atmosphere of speech or text in a conversation, and is an element used to convey emotion and intention.

[1038] "Adjusting the content of the dialogue" means changing the content and expression of the message sent by the avatar depending on the user's emotional state.

[1039] This invention is a system that collects life log data and characteristic data such as sentences and voiceprints from the messenger apps and internet search tools that users use daily, and recreates the user's memories and personality based on this data.Furthermore, by combining it with an emotion engine, it realizes dialogue that corresponds to the user's emotional state.

[1040] Data collection

[1041] When users use messaging apps (e.g., WhatsApp) or internet search tools (e.g., Google Search), life log data (e.g., chat history, search history) and characteristic data such as sentences and voiceprints are collected from these applications. Data collection is performed using a script that retrieves data through an API.

[1042] Data Preprocessing

[1043] The server receives the collected data and filters out unnecessary information. For example, it removes unnecessary advertising messages from chat history and extracts only user comments. This process is performed using the Python pandas library.

[1044] Data analysis and feature extraction

[1045] The server analyzes the preprocessed data and extracts features such as the user's behavioral patterns, hobbies, interests, and personality. For example, natural language processing (NLP) techniques are used to extract emotions and topics from the user's comments. Libraries such as NLTK and spaCy are used for this analysis.

[1046] Training an AI model

[1047] The server uses the extracted feature data as training data to train a generative AI model (e.g., GPT-4). This gives the model the ability to reproduce the user's memories and personality. Frameworks such as TensorFlow and PyTorch are used for training.

[1048] emotion recognition

[1049] The server uses an emotion engine (e.g., IBM Watson's emotion analysis API) to recognize emotions from the user's text or voice in real time. For example, if a user sends a message saying, "I had so much fun today!", the emotion engine will recognize the emotion "fun."

[1050] Avatar Reaction Adjustment

[1051] The device adjusts the tone and dialogue of the avatar based on information from the emotion engine. For example, if the user is happy, the avatar will respond in a cheerful tone, saying, "That's great! What happened?" Conversely, if the user is sad, the avatar will respond in a calmer tone, saying, "That must be hard. Is there anything I can do to help?"

[1052] Examples and prompts

[1053] Specific examples

[1054] If a user sends a message in a messenger app saying, "I had so much fun today!", the server receives this message and recognizes the emotion "fun" through the emotion engine. Based on this information, the device's avatar responds in a cheerful tone, saying, "That's great! What happened?"

[1055] Prompt Sentence Examples

[1056] "Design a system that analyzes the emotions of messages sent by users on a messenger app and has an avatar respond in an appropriate tone. Specifically, if the user is happy, the avatar should respond in a brighter tone, and if the user is sad, the avatar should respond in a calmer tone."

[1057] In this way, a system is constructed that realizes a dialogue according to the emotional state of the user.

[1058] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1059] Step 1:

[1060] When users use messenger apps or internet search tools, life log data and characteristic data such as sentences and voiceprints are collected from these applications.

[1061] Input: User chat history, search history, and voice data

[1062] Output: Collected life log data, text data, and voiceprint data

[1063] Specific operation: Executes a script that retrieves data through the API and saves it in the database.

[1064] Step 2:

[1065] The server receives the collected data and filters out unnecessary information.

[1066] Input: Collected life log data, text data, voiceprint data

[1067] Output: Preprocessed data (data with unnecessary information removed)

[1068] Specific operation: Using Python's pandas library, create a data frame and remove unnecessary rows and columns.

[1069] Step 3:

[1070] The server analyzes the preprocessed data and extracts characteristics such as the user's behavioral patterns, hobbies, interests, and personality.

[1071] Input: Preprocessed data

[1072] Output: User characteristic data (behavioral patterns, hobbies, interests, personality, etc.)

[1073] What it does: Extract sentiment and topics from text data using natural language processing (NLP) techniques, using libraries such as NLTK and spaCy.

[1074] Step 4:

[1075] The server uses the extracted feature data as training data to train the generative AI model.

[1076] Input: User feature data

[1077] Output: A trained generative AI model

[1078] What it does: Train generative AI models (e.g., GPT-4) using frameworks like TensorFlow and PyTorch.

[1079] Step 5:

[1080] The server uses an emotion engine to recognize emotions from the user's text and voice in real time.

[1081] Input: User's text data, voice data

[1082] Output: Recognized emotion data

[1083] Specific operation: Calls an emotion engine (e.g., IBM Watson's emotion analysis API) and analyzes emotions.

[1084] Step 6:

[1085] The device adjusts the tone and dialogue of the avatar based on information from the emotion engine.

[1086] Input: Recognized emotion data

[1087] Output: Adjusted dialogue

[1088] Specific behavior: Based on the emotion data, the avatar generates dialogue content and responds to the user. For example, if the user is happy, the avatar responds in a cheerful tone, saying, "That's great! What happened?"

[1089] (Application example 1)

[1090] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1091] Conventional content distribution services have had the problem of not being able to adequately provide personalized content recommendations based on the user's emotional state or individual hobbies and interests. It was also difficult to grasp the user's emotional state in real time and provide appropriate content accordingly.

[1092] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., a means for AI to reproduce the user's memories and personality using the data as training data, a means for generating an avatar of the user on the cloud, a means including an emotion engine that grasps the user's emotional state in real time, and a means for recommending optimal content according to the user's emotional state. This enables personalized content recommendations according to the user's emotional state and individual hobbies and interests.

[1093] A "messenger app" is an application that allows users to send and receive text messages, voice messages, images, videos, etc.

[1094] An "Internet search tool" is software or a web service that allows users to search for information on the Internet.

[1095] "Life log data" is data recorded about a user's daily life behavior and activities.

[1096] "Feature data" is data that represents individual characteristics of a user, such as their writing or voiceprint.

[1097] "Training data" is a dataset used to train a machine learning model.

[1098] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence.

[1099] "Means of recreating the user's memories and personality" refers to technology that recreates a user's past behavior and personality based on collected data.

[1100] The "cloud" is a collection of computer resources and services provided over the Internet.

[1101] An "avatar" is a digital avatar that acts as a user's alter ego.

[1102] The "emotion engine" is a technology that recognizes the user's emotional state in real time and generates a corresponding response.

[1103] "Content recommendation" is a technology that suggests optimal content based on a user's hobbies, interests, and emotional state.

[1104] A system for implementing this invention is configured as follows: First, life log data and characteristic data such as text and voiceprints are collected from messenger apps and internet search tools installed on a user's smartphone or personal computer. This data reflects the user's behavioral patterns, hobbies, interests, personality, etc.

[1105] The server uses the collected life log data and feature data as training data and uses a generative AI model to recreate the user's memories and personality. This generative AI model is implemented using a programming language such as Python and runs on the cloud. A digital avatar of the user is generated on the cloud, enabling interactions that reflect the user's memories and personality.

[1106] In addition, the app is equipped with an emotion engine that understands the user's emotional state in real time. This emotion engine analyzes the user's text and voice messages to recognize their emotional state. For example, if a user types "I'm a little tired today," the emotion engine will detect the user's sense of fatigue.

[1107] The server then recommends optimal content to the user based on the emotional state obtained from the emotion engine. Content recommendations are based on the user's behavioral patterns and emotional state, and suggest movies, music, and other content that will help the user relax.

[1108] For example, if a user types "I feel a bit tired today," the emotion engine will detect the user's sense of fatigue and recommend relaxing movies and music. This process is achieved by inputting the following prompt sentence into the generative AI model:

[1109] "Analyze the user's emotional state and recommend relaxing content. The user's input is 'I'm a little tired today.'"

[1110] In this way, personalized content recommendations based on the user's emotional state and individual hobbies and interests become possible.

[1111] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1112] Step 1:

[1113] Users use messenger apps and internet search tools installed on their smartphones or personal computers to send messages and perform searches on a daily basis. This generates lifelog data and feature data such as sentences and voiceprints. The input is the user's messages and search queries, and the output is the lifelog data and feature data.

[1114] Step 2:

[1115] The device transmits the generated life log data and feature data to the server. The input is the life log data and feature data, and the output is the data transmitted to the server. Specifically, the device performs data collection and transmission.

[1116] Step 3:

[1117] The server inputs the received life log data and feature data into a generative AI model as training data to recreate the user's memories and personality. The input is life log data and feature data, and the output is a model that recreates the user's memories and personality. Specific operations include data preprocessing, model training, and model generation.

[1118] Step 4:

[1119] The server generates a digital avatar of the user on the cloud. The input is a model that recreates the user's memories and personality, and the output is a digital avatar on the cloud. Specifically, the avatar is generated and placed on the cloud.

[1120] Step 5:

[1121] When a user inputs a message, the terminal sends the message to the server. The input is the user's message, and the output is the message sent to the server. The specific operation is to send the message.

[1122] Step 6:

[1123] The server inputs the received message into the emotion engine and grasps the user's emotional state in real time. The input is the user's message, and the output is the user's emotional state. Specifically, the engine analyzes the message and recognizes the user's emotional state.

[1124] Step 7:

[1125] The server recommends optimal content based on the emotional state obtained from the emotion engine. The input is the user's emotional state, and the output is the recommended content. Specifically, the server evaluates the emotional state and selects content.

[1126] Step 8:

[1127] The server sends the recommended content to the terminal and presents it to the user. The input is the recommended content, and the output is the content presented to the user. Specifically, the content is sent and displayed.

[1128] In this way, personalized content recommendations based on the user's emotional state and individual hobbies and interests are realized.

[1129] Example 2

[1130] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1131] In modern society, users are often so busy that they neglect communication with acquaintances, friends, and family. Furthermore, even after a user passes away, there is a need for those left behind to continue to communicate with them. However, with conventional technology, it has been difficult to accurately reproduce a user's language and thought patterns, and to engage in appropriate conversations based on their emotional state.

[1132] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1133] In this invention, the server includes means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., means for artificial intelligence to use this data as training data to reproduce the user's memories and personality, means for generating an avatar of the user on the cloud, means for analyzing the user's emotional state, means for generating dialogue for the avatar based on the results of the emotion analysis, and means for executing the generated dialogue. This makes it possible for the user to maintain communication with acquaintances, friends, and family even when they are absent or busy, or even after they have passed away.

[1134] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[1135] An "Internet search tool" is software that allows users to search for information on the Internet and obtain the data they need.

[1136] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, health data, etc.

[1137] "Characteristic data" is data that represents a user's individual characteristics, including writing style and voiceprint.

[1138] "Artificial intelligence" is the technology that enables computers to mimic human intelligence and learn, reason, and self-correct.

[1139] The "cloud" is a collection of computer resources and services provided over the Internet.

[1140] An "avatar" is a virtual entity that mimics the user's speech and thought patterns and converses on the user's behalf.

[1141] An "emotional state" is a state that indicates the user's current emotion, and includes joy, anger, sadness, etc.

[1142] "Sentiment analysis" is the process of reading and analyzing emotions from a user's text data or voice data.

[1143] "Dialogue generation" is the process of generating appropriate responses based on the user's emotional state and past dialogue history.

[1144] "Dialogue execution" is the process of actually sending the generated dialogue on behalf of the user.

[1145] This invention is a system that generates a user's avatar on the cloud and allows the user to maintain communication with acquaintances, friends, and family even when the user is absent, busy, or has passed away. Specific embodiments of this system are described below.

[1146] First, the server collects user data. Specifically, it acquires life log data and characteristic data such as sentences and voiceprints from messenger apps and internet search tools. This data is used as training data to learn the user's vocabulary and thought patterns.

[1147] The server then analyzes the collected data and learns the user's language and thought patterns. Specifically, it analyzes the data using natural language processing technology and uses a generative AI model (such as Google Cloud Natural Language API or OpenAI's GPT-4) to learn the user's characteristics.

[1148] The server then generates an avatar of the user based on the learned model. This avatar mimics the user's language and thought patterns and interacts on the user's behalf. The avatar is placed in the cloud and sends and receives messages on the user's behalf.

[1149] The server then uses an emotion engine to analyze the user's emotional state. Specifically, it analyzes the user's text and voice data to understand their emotional state in real time. For example, it uses Microsoft Azure's Text Analytics for Sentiment Analysis to perform emotion analysis.

[1150] Based on the emotion analysis results, the server generates dialogue for the avatar. For example, if the user is angry, it will choose words of apology, and if the user is happy, it will choose words of congratulations. The generated dialogue will be appropriate depending on the user's emotional state.

[1151] Finally, the device executes the generated dialogue and sends messages on behalf of the user, allowing users to maintain communication with acquaintances, friends, and family even when they are away or busy.

[1152] As a specific example, if a user is too busy to reply, the avatar will send a message on their behalf. For example, if the user receives a message from a friend asking, "What are your plans for this weekend?", the avatar will reply, "I'm planning to spend this weekend with my family," based on the user's past conversation history. Also, if the user is angry, the emotion engine will detect that emotion and the avatar will choose words of apology. For example, if the user sends an angry message saying, "Why did this happen?", the avatar will reply, "I'm sorry, I'll deal with it right away."

[1153] An example of a prompt sentence might be:

[1154] "Based on the user's past messaging history, generate an appropriate reply to the following message: 'What are your plans this weekend?'"

[1155] "Generate an appropriate response for when a user is angry. User message: 'Why did this happen to me?'"

[1156] In this way, the server and the terminal work together to generate an avatar of the user and use the emotion engine to carry out appropriate dialogue according to the user's emotional state.

[1157] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1158] Step 1:

[1159] The server collects user data. Specifically, it acquires life log data and feature data such as sentences and voiceprints from messenger apps and internet search tools. User account information and access permissions are required as input, and the collected data is obtained as output. For example, email data is collected using the Gmail API.

[1160] Step 2:

[1161] The server analyzes the collected data and learns the user's language usage and thought patterns. Specifically, it uses natural language processing technology to analyze the data and uses a generative AI model (for example, Google Cloud Natural Language API or OpenAI's GPT-4). The collected data is required as input, and a model that captures the user's characteristics is generated as output. For example, past message history can be analyzed to extract the user's language usage patterns.

[1162] Step 3:

[1163] The server generates an avatar for the user based on the trained model. The avatar imitates the user's language and thought patterns and converses on behalf of the user. The trained model is required as input, and an avatar deployed on the cloud is generated as output. For example, an avatar for the user is created and deployed on the cloud.

[1164] Step 4:

[1165] The server uses an emotion engine to analyze the user's emotional state. Specifically, it analyzes the user's text data and voice data to grasp their emotional state in real time. The input requires the user's latest messages and voice data, and the output is the emotion analysis results. For example, emotion analysis can be performed using Microsoft Azure's Text Analytics for Sentiment Analysis.

[1166] Step 5:

[1167] The server generates the avatar's dialogue based on the emotion analysis results. For example, if the user is angry, it chooses words of apology, and if the user is happy, it chooses words of congratulations. The input requires the emotion analysis results and a model that captures the user's characteristics, and the output is the generated dialogue. For example, if the user sends an angry message such as "Why did this happen?", the avatar will reply, "I'm sorry, I'll deal with it right away."

[1168] Step 6:

[1169] The device executes the generated dialogue and sends messages on behalf of the user. It takes the generated dialogue as input and the sent message as output. For example, if the user's friend sends a message asking "What are your plans this weekend?", the avatar replies "I'm planning to spend this weekend with my family."

[1170] In this way, the server and the terminal work together to generate an avatar of the user and use the emotion engine to carry out appropriate dialogue according to the user's emotional state.

[1171] (Application example 2)

[1172] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1173] In modern society, users are often too busy to fully utilize content distribution services, and there are also issues with not being able to communicate appropriately when users are away. There is also the issue of it being difficult to respond appropriately to users' emotional states. Furthermore, there is a lack of ways to maintain communication with acquaintances, friends, and family members even after a user passes away.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1175] In this invention, the server includes a means for linking life log data from messenger apps, internet search tools, etc. with characteristic data such as text and voiceprints; a means for AI to use this data as training data to recreate the user's memories and personality; a means for generating an avatar of the user on the cloud; a means for recommending content based on the user's viewing history and ratings; a means for posting comments that imitate the user's language and thought patterns; and a means for controlling the avatar's behavior according to the user's emotional state. This allows users to fully utilize content distribution services even when they are busy, enables appropriate communication even when the user is absent, and enables appropriate responses according to the user's emotional state. Furthermore, it allows users to maintain communication with acquaintances, friends, and family even after their death.

[1176] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[1177] An "Internet search tool" is software that allows users to search for information on the Internet.

[1178] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, health data, etc.

[1179] "Characteristic data" is data used to identify an individual, such as a user's writing or voiceprint.

[1180] "Teacher data" is data used to train a machine learning model, and consists of pairs of input data and corresponding correct answer data.

[1181] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[1182] "Means of recreating a user's memories and personality" refers to technology that imitates a user's thought patterns and language usage based on the user's past actions and statements.

[1183] "Means for generating a person's avatar on the cloud" refers to a technology that creates a digital avatar of a user on a server accessible via the Internet.

[1184] "Viewing history" is a record of content that a user has viewed in the past.

[1185] "Ratings" are records of ratings and feedback given by users to content they have viewed.

[1186] "Means for recommending content" refers to technology that suggests new content that may interest users based on their viewing history and ratings.

[1187] "Method of posting comments that imitate language and thought patterns" refers to a technology that generates and posts comments that a user is likely to make based on the user's past comments.

[1188] "Means for controlling the behavior of the avatar according to the emotional state" refers to technology that analyzes the user's emotions and adjusts the avatar's behavior and speech based on the results.

[1189] An embodiment of the present invention will be described. First, the entire system is built on the cloud and operates in conjunction with the user's device (smartphone or head-mounted display). The system is composed of the following main components:

[1190] 1. Data Collection

[1191] The server collects life log data and feature data (such as text and voiceprints) from messenger apps and internet search tools. This data includes the user's behavioral history and what they say.

[1192] 2. Incarnation generation

[1193] The server uses the collected data as training data to train a generative AI model (e.g., GPT-4). This model is used to recreate the user's memories and personality, specifically, mimicking the user's speech and thought patterns.

[1194] 3. Cloud Avatar

[1195] The server creates an avatar of the user on the cloud. This avatar communicates on the user's behalf when the user is absent or busy. It also maintains communication with acquaintances, friends, and family even after the user's death.

[1196] 4. Content Recommendation

[1197] The server uses a generative AI model to recommend new content based on the user's viewing history and ratings, for example, suggesting the next movie to watch based on the genre of movies the user has recently watched.

[1198] Prompt Sentence Examples

[1199] "The genre of movies the user recently watched was action. Please recommend three movies that would be good next."

[1200] 5. Post a comment

[1201] The server mimics the user's language and thought patterns and posts comments on their behalf when they are absent or busy, allowing them to maintain their presence.

[1202] Prompt Sentence Examples

[1203] "Generate comments that users post after watching 'Inception'."

[1204] 6. Emotional Response

[1205] The server uses an emotion engine (e.g., Emotion API) to analyze the user's emotional state. Based on the analysis results, the avatar's behavior and speech are adjusted. For example, if the user is sad, it posts encouraging words, and if the user is happy, it posts congratulatory words.

[1206] In this way, users can fully utilize content distribution services even when they are busy, and appropriate communication can be maintained even when the user is absent. It also enables appropriate responses based on the user's emotional state, and allows communication with acquaintances, friends, and family to continue even after the user's death.

[1207] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1208] Step 1:

[1209] The server collects life log data and feature data (such as text and voiceprints) from messenger apps and internet search tools. The input is the user's behavioral history and comments, and the output is to store this data on the cloud. Specifically, the data is obtained through an API and stored in an AWS S3 bucket.

[1210] Step 2:

[1211] The server uses the collected data as training data to train a generative AI model (e.g., GPT-4). The input is the collected life log data and feature data, and the output is a model for reproducing the user's memories and personality. Specifically, the data is preprocessed and fed into the AI ​​model for learning.

[1212] Step 3:

[1213] The server generates a user avatar on the cloud. The input is a trained AI model, and the output is a digital avatar of the user. Specifically, it deploys the AI ​​model to the server on the cloud and provides an API to generate the user avatar.

[1214] Step 4:

[1215] The server uses a generative AI model to recommend new content based on the user's viewing history and ratings. The input is the user's viewing history and ratings, and the output is the recommended content. Specifically, the server analyzes the viewing history and ratings and inputs prompts into the AI ​​model to generate recommendation results.

[1216] Step 5:

[1217] The server imitates the user's language and thought patterns and posts comments on their behalf when they are absent or busy. The input is the user's past comment data, and the output is the generated comment. Specifically, it analyzes the past comment data and inputs prompt sentences into the AI ​​model to generate comments.

[1218] Step 6:

[1219] The server uses an emotion engine (e.g., Emotion API) to analyze the user's emotional state. The input is the user's current emotional data, and the output is the analysis result. Specifically, the server acquires emotional data in real time and analyzes it using the Emotion API.

[1220] Step 7:

[1221] The server adjusts the avatar's behavior and speech based on the analysis results. The input is the emotion engine's analysis results, and the output is the adjusted avatar's behavior and speech. Specifically, it inputs prompt statements into the AI ​​model based on the analysis results, generating appropriate behavior and speech.

[1222] Example 3

[1223] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1224] Previous systems had low accuracy in reproducing the user's memories and personality, and had difficulty responding in real time to the user's emotional state. Furthermore, they lacked long-term data collection and analysis, making it impossible to accurately reflect the user's emotional state. This limited the user experience and reduced the system's usefulness.

[1225] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a means for linking life log data and feature data such as text and voiceprints from a messenger app, an internet search tool, etc., a means for an AI to reproduce the user's memory and personality using the data as training data, a means for generating an avatar of the user on the cloud, a means for detecting the user's emotional state in real time, a means for controlling the avatar's facial expression according to the user's emotional state, and a means for increasing the degree of reproduction the longer the monthly subscription fee is paid. This makes it possible to reproduce the user's memory and personality with high accuracy and to react in real time according to the emotional state.

[1226] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc. in real time.

[1227] An "Internet search tool" is software that allows users to search for information on the Internet and obtain the data they need.

[1228] "Life log data" refers to data that includes behavioral history and activity records related to a user's daily life.

[1229] "Characteristic data" is unique data used to identify an individual, such as a user's writing or voiceprint.

[1230] "Teacher data" is known input-output pair data used to train a machine learning model.

[1231] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence to learn, reason, and self-correct.

[1232] The "cloud" is a collection of computer resources and services provided over the Internet.

[1233] An "avatar" is a virtual being that reproduces the user's memories and personality and interacts and acts on the user's behalf.

[1234] An "emotional state" is a state that indicates the user's current feelings or mood.

[1235] "Real-time" refers to the instantaneous collection, analysis, and reaction of data.

[1236] A "monthly subscription" is a pricing system in which users pay a fixed monthly fee to use a service.

[1237] "Reproducibility" refers to the degree to which the system can accurately reproduce the user's memories and personality.

[1238] MODE FOR CARRYING OUT THE INVENTION

[1239] This invention is a system that reproduces the user's memories and personality with high accuracy and responds in real time according to the user's emotional state. Specific embodiments of this system are described below.

[1240] 1. Program Generation

[1241] The server collects user data and uses a generative AI model to generate a program that recreates the user's memories and personality, including an emotion engine that controls the avatar's facial expressions according to the user's emotional state.

[1242] 2. Program processing explanation

[1243] As users continue to pay their monthly subscription, the server collects more data, including their behavioral history, emotional state, and conversational content. The collected data is analyzed using a generative AI model (e.g., OpenAI's GPT-4) and used to recreate the user's memories and personality.

[1244] The device uses hardware such as a camera and microphone to detect the user's emotional state in real time. For example, the camera captures the user's facial expressions, and the microphone analyzes the user's tone of voice. This data is sent to an emotion engine to determine the user's emotional state.

[1245] The emotion engine controls the facial expression of the avatar based on the user's emotional state. For example, when the user is happy, the emotion engine makes the avatar smile. When the user is sad, the emotion engine makes the avatar sad.

[1246] 3. Examples of concrete examples and prompts

[1247] As a concrete example, consider the case where a user has been paying for a service for a year. Over that time, the system has collected a huge amount of data about the user's behavioral history and emotional state. Based on this data, the generative AI model can reproduce the user's memories and personality with a very high degree of accuracy.

[1248] Example prompt sentence:

[1249] Generate a program to recreate the user's memories and personality based on data from a user who has continued paying for a year. Also, configure the emotion engine so that the avatar smiles when the user is happy and sad when the user is sad.

[1250] In this way, the server collects data and uses the generative AI model to generate a program that reproduces the user's memories and personality. The terminal detects the user's emotional state, and the emotion engine controls the avatar's facial expressions. As the user continues to pay over a long period of time, the system collects more data, and the avatar's reproduction becomes more accurate. The flow of the identification process in Example 3 is explained using Figure 21.

[1251] Program processing flow

[1252] Step 1: Collect user data

[1253] Users log in to the system and engage in everyday activities and interactions. The device uses a camera and microphone to capture the user's facial expressions and tone of voice in real time, and also collects text data entered by the user and their behavioral history.

[1254] Input: User's facial expression data, tone of voice, text data, behavioral history

[1255] Output: Collected user data

[1256] Specific behavior:

[1257] The device's camera captures the user's face and collects facial expression data.

[1258] The device's microphone records the user's voice and analyzes the tone and emotion of the voice.

[1259] Collect text data entered by users in chat.

[1260] Step 2: Sending data to the server

[1261] The device periodically sends the collected data to a server, where it is encrypted and securely transmitted.

[1262] Input: Collected user data

[1263] Output: Data sent to the server

[1264] Specific behavior:

[1265] The data collected by the device is encrypted and sent to the server.

[1266] The server stores the received data in a database.

[1267] Step 3: Data analysis on the server

[1268] The server analyzes the received data to identify the user's behavioral patterns and emotional state using data mining techniques and machine learning algorithms.

[1269] Input: Data sent to the server

[1270] Output: Analysis results (user behavior patterns, emotional state)

[1271] Specific behavior:

[1272] The server retrieves the data from the database and applies an analysis algorithm.

[1273] The server identifies the user's behavioral patterns and emotional state and stores the results.

[1274] Step 4: Recreate user memories and personality with a generative AI model

[1275] The server uses a generative AI model (e.g., OpenAI's GPT-4) based on the analysis results to recreate the user's memories and personality. The generated model then simulates the user's interactions and behavior.

[1276] Input: Analysis results

[1277] Output: Recreated user's memories and personality

[1278] Specific behavior:

[1279] The server inputs the analysis results into a generative AI model to recreate the user's memories and personality.

[1280] The generated model simulates user interactions and behavior.

[1281] Step 5: Detecting the user's emotional state with the emotion engine

[1282] The device uses an emotion engine to detect the user's emotional state in real time. The emotion engine analyzes data from the camera and microphone to determine the user's emotional state.

[1283] Input: Real-time data from camera and microphone

[1284] Output: User's emotional state

[1285] Specific behavior:

[1286] The device's emotion engine analyzes facial expression data from the camera to determine the user's emotional state.

[1287] The device's emotion engine analyzes the tone of voice from the microphone to determine the user's emotional state.

[1288] Step 6: Avatar Facial Expression Control

[1289] After the emotion engine determines the user's emotional state, the device controls the avatar's facial expression. For example, if the user is happy, the avatar will smile. If the user is sad, the avatar will make a sad expression.

[1290] Input: User's emotional state

[1291] Output: Avatar facial expression

[1292] Specific behavior:

[1293] The device controls the avatar's facial expressions based on the judgment results of the emotion engine.

[1294] The avatar makes appropriate facial expressions depending on the user's emotional state.

[1295] (Application example 3)

[1296] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1297] Conventional avatar systems focus on recreating the user's memories and personality, but they have the drawback of being unable to provide real-time responses or personalized content based on the user's emotional state. Furthermore, they lack the means to analyze the user's emotional state, which can lead to a decline in the quality of the user experience.

[1298] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., means for AI to reproduce the user's memories and personality using these as training data, means for generating an avatar of the user on the cloud, means for analyzing the user's emotional state in real time, and means for providing personalized content according to the analyzed emotional state. This makes it possible to provide real-time responses and personalized content according to the user's emotional state.

[1299] A "messenger app" is software that allows users to send and receive text and voice messages.

[1300] An "Internet search tool" is software for searching for information on the Internet.

[1301] "Life log data" refers to data related to a user's daily life, including behavioral history and location information.

[1302] "Characteristic data" is data that indicates the individual characteristics of a user, and includes sentences, voiceprints, and the like.

[1303] "Training data" is data used to train a machine learning model.

[1304] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence.

[1305] "Reproducing the user's memories and personality" means imitating the user's personality and thought patterns based on the user's past actions and statements.

[1306] "Creating an avatar of the user on the cloud" means creating a virtual avatar of the user on a server accessible via the Internet.

[1307] "Analyzing the user's emotional state in real time" means instantly determining the user's emotions at any given time based on their facial expressions, voice, etc.

[1308] "Providing personalized content" means providing optimal information and entertainment based on a user's individual preferences and emotional state.

[1309] A system for implementing the present invention uses the following hardware and software to provide personalized content according to the user's emotional state.

[1310] Hardware

[1311] 1. Camera: Used to capture the user's face. This uses the camera built into the smartphone or head-mounted display.

[1312] 2. Server: Use cloud servers to process and store data.

[1313] software

[1314] 1. OpenCV: A library for face detection. It analyzes images captured by a camera and detects faces.

[1315] 2. Keras: A deep learning library for implementing emotion recognition models. It predicts user emotions from detected facial images.

[1316] 3. Requests: A library for communicating with content delivery servers. It is used to obtain content according to the user's emotional state.

[1317] Data processing and calculation

[1318] 1. Data capture: Capture video from the camera in real time and perform face detection. Detect faces using OpenCV, convert the detected faces to grayscale, and adjust their size.

[1319] 2. Emotion prediction: The adjusted facial image is input into the Keras emotion recognition model to predict the user's emotion. There are seven types of emotions: "anger," "disgust," "fear," "happiness," "sadness," "surprise," and "neutral."

[1320] 3. Content Serving: Based on the predicted emotion, the Requests library is used to send a request to the content delivery server to retrieve the appropriate content, which is then displayed to the user.

[1321] Specific examples

[1322] For example, when a user is using a smartphone, if the camera captures the user's face and recognizes the emotion as "happy," the content distribution server will provide "comedy movies" or "fun music playlists."

[1323] Prompt Sentence Examples

[1324] "Develop an application that analyzes user emotions in real time and provides personalized content based on those emotions. Use a Keras model for emotion recognition and the Requests library for communication with the content delivery server."

[1325] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1326] Step 1:

[1327] The device uses a camera to capture the user's face in real time. The input is video data from the camera, and the output is the captured image frame. Specifically, the device starts the camera and continuously captures video.

[1328] Step 2:

[1329] The device uses OpenCV to detect faces in captured image frames. The input is the captured image frame, and the output is the detected face area. Specifically, the device converts the image to grayscale and applies a face detection algorithm to identify the face location.

[1330] Step 3:

[1331] The device converts the detected face area to grayscale and resizes it to 48x48 pixels. The input is the detected face area, and the output is the resized face image. Specifically, the device normalizes the face image and converts it into a format suitable for machine learning models.

[1332] Step 4:

[1333] The device predicts emotions from resized facial images using a Keras emotion recognition model. The input is the resized facial image, and the output is the predicted emotion label. Specifically, the device inputs the facial image into the model, calculates the probability distribution of emotions, and selects the emotion with the highest probability.

[1334] Step 5:

[1335] The device sends a request to the content delivery server using the Requests library based on the predicted emotion label. The input is the predicted emotion label, and the output is the content obtained from the content delivery server. Specifically, the device sends the emotion label in JSON format to the server and requests the appropriate content.

[1336] Step 6:

[1337] The server selects personalized content based on the received emotion label and sends it back to the device. The input is the emotion label and the output is the selected content. Specifically, the server searches the database for content corresponding to the emotion label and sends it to the device.

[1338] Step 7:

[1339] The terminal displays the content received from the server to the user. The input is the content received from the server, and the output is the content displayed to the user. In concrete terms, the terminal displays the received content on the screen so that the user can view or use it.

[1340] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1341] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of a data generation model58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1342] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1343] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1344] [Third embodiment]

[1345] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1346] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1347] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1348] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1349] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1350] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1351] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1352] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1353] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1354] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1355] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1356] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1357] "Example 1"

[1358] In one embodiment of the present invention, life log data and characteristic data such as text and voiceprints are collected from the user's daily use of messenger apps, internet search tools, etc. This data reflects the user's behavioral patterns, hobbies, interests, personality, etc. By using this data as training data and training AI, the user's memories and personality are reproduced.

[1359] "Example 2"

[1360] Next, an avatar of the user is generated on the cloud. This avatar is designed to maintain communication with acquaintances, friends, and family when the user is absent, busy, or even after the user has passed away. Specifically, the avatar mimics the user's speech and thought patterns, recreating how the user normally interacts.

[1361] "Example 3"

[1362] Furthermore, the system of the present invention employs a monthly subscription system. The longer a user continues to pay, the more data the system collects and the more the AI ​​learns. This allows the avatar to be more accurately recreated, and the user's memories and personality to be more accurately reproduced.

[1363] The processing flow of each embodiment will be described below.

[1364] "Example 1"

[1365] Step 1: Collect life log data and characteristic data such as text and voiceprints from messenger apps, internet search tools, etc. that users use on a daily basis.

[1366] Step 2: The collected data is used as training data for the AI ​​to learn. Through this learning, the AI ​​will understand the user's behavioral patterns, hobbies, interests, personality, etc.

[1367] Step 3: As the AI ​​learns, it becomes better at replicating the user's memories and personality.

[1368] "Example 2"

[1369] Step 1: Create an avatar of the user on the cloud. This avatar will maintain communication with acquaintances, friends, and family when the user is absent, busy, or even after the user has passed away.

[1370] Step 2: The avatar mimics the user's language and thought patterns, replicating how the user normally interacts.

[1371] "Example 3"

[1372] Step 1: The system of the present invention employs a monthly billing system.

[1373] Step 2: The longer the user continues to pay, the more data the system collects and the more the AI ​​learns.

[1374] Step 3: This increases the fidelity of the avatar, allowing it to more accurately recreate the user's memories and personality.

[1375] Example 1

[1376] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1377] Modern society demands technology that can collect data reflecting a user's behavioral patterns, interests, and personality, and then use that data to recreate their memories and personality. However, existing systems lack integration between the steps of data collection, preprocessing, storage, learning, evaluation, and generation, making it difficult to efficiently recreate a user's memories and personality. Furthermore, there is a lack of ways for users to maintain communication with acquaintances, friends, and family, even when they are absent or have passed away. Furthermore, there is no established sustainable billing model for improving the level of recreation.

[1378] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1379] In this invention, the server includes: means for collecting life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc.; means for preprocessing the collected data and converting it into a format suitable for learning an AI model; means for saving the preprocessed data in a database; means for inputting the saved data as training data into an AI model and training it to reproduce the user's memories and personality; means for evaluating the AI ​​model after training and confirming its performance; means for saving well-rated models; and means for generating an avatar of the user on the cloud. This allows for efficient collection and processing of data reflecting the user's behavioral patterns, interests, and personality, enabling highly accurate reproduction of the user's memories and personality. Furthermore, the system allows the user to maintain communication with acquaintances, friends, and family even when they are absent or after their death, and the level of reproduction can be improved through a sustainable billing model.

[1380] A "messenger app" is a software application that enables users to send and receive text messages, voice messages, images, videos, etc.

[1381] An "Internet search tool" is a software application that allows a user to search for information on the Internet.

[1382] "Life log data" is data that records a user's actions and activities in their daily life.

[1383] "Characteristic data" is data that reflects personal characteristics, such as a user's writing style or voiceprint.

[1384] "Preprocessing" is the process of converting collected data into a format suitable for learning AI models.

[1385] A "database" is a system for efficiently storing, managing, and retrieving data.

[1386] "Training data" is data with correct labels that is used to train an AI model.

[1387] An "AI model" is a mathematical model that uses artificial intelligence technology to analyze data and make predictions and classifications.

[1388] "Learning" is the process by which an AI model discovers patterns and rules based on training data.

[1389] "Evaluation" is the process of checking the performance of an AI model after it has completed training.

[1390] The "cloud" is a collection of computer resources and services provided over the Internet.

[1391] An "avatar" is a digital being that recreates the user's memories and personality.

[1392] A "charging model" is a business model for collecting fees for the use of a service.

[1393] MODE FOR CARRYING OUT THE INVENTION

[1394] This invention is a system that collects life log data and characteristic data such as sentences and voiceprints from messenger apps and internet search tools that users use on a daily basis, and recreates the user's memories and personality based on this data. Specific embodiments of this system are described below.

[1395] Data collection

[1396] The server collects life log data and characteristic data such as sentences and voiceprints from the messenger apps and internet search tools that users use on a daily basis. This data is collected using common software such as messenger apps and internet search tools. The collected data reflects the user's behavioral patterns, hobbies, interests, personality, etc.

[1397] Data Preprocessing

[1398] The server preprocesses the collected data and converts it into a format suitable for learning AI models. Specifically, it tokenizes text data and converts audio data into a spectrogram. This is done using Python libraries such as NLTK and Librosa. For example, NLTK is used to tokenize text data, and Librosa is used to convert audio data into a spectrogram.

[1399] Data storage

[1400] The server stores the preprocessed data in a database, for example using a database system such as MySQL or MongoDB, by establishing a database connection and inserting the preprocessed data into the appropriate tables and collections.

[1401] Training an AI model

[1402] The server inputs the preprocessed data into the AI ​​model as training data, and trains it to reproduce the user's memories and personality. This training uses generative AI models such as GPT-4 and BERT. Specifically, it initializes the model, batches the data, and executes the training.

[1403] Evaluating and saving the model

[1404] The server evaluates the AI ​​model after training and checks its performance. Evaluation uses metrics such as precision and recall. Models that pass the evaluation are stored in a file system or cloud storage.

[1405] Avatar generation on the cloud

[1406] The server uses the well-reviewed model to generate a cloud-based avatar of the user, which recreates the user's memories and personality and is used to maintain communication with acquaintances, friends, and family.

[1407] Specific examples

[1408] Example 1: Data collection from messenger apps

[1409] The server collects messages sent and received by users on popular messaging apps. These messages reflect the user's conversation patterns and interests. This message data is preprocessed and trained into an AI model to reproduce the user's conversation style.

[1410] Example 2: Data collection from internet search tools

[1411] The server collects search queries performed by users using common internet search tools. The collected search queries reflect the user's interests and concerns. This search data is preprocessed and trained into an AI model to reproduce the user's interests.

[1412] Prompt Sentence Examples

[1413] "Create a program to collect messages sent by users on popular messaging apps and train an AI model to replicate the user's conversational style."

[1414] "You will create a program that collects search queries performed by users on popular internet search tools and trains an AI model to replicate the user's interests."

[1415] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1416] Step 1: Data collection

[1417] The server collects life log data and feature data such as sentences and voiceprints from the messenger apps and internet search tools that users use on a daily basis. It receives the log data from the messenger apps and internet search tools as input and generates the collected raw data as output. Specifically, the server obtains data using APIs and scraping technology, and collects the data using secure communication protocols (e.g., HTTPS).

[1418] Step 2: Data Preprocessing

[1419] The server preprocesses the collected data and converts it into a format suitable for training the AI ​​model. It receives the collected raw data as input and produces preprocessed data as output. Specifically, the server performs the following operations:

[1420] Tokenizing text data: Splitting sentences into words and phrases using Python's NLTK library.

[1421] Spectrogram conversion of audio data: Use the Librosa library in Python to convert audio data into a frequency spectrum.

[1422] Step 3: Save your data

[1423] The server stores the preprocessed data in a database. It receives the preprocessed data as input and generates the data stored in the database as output. Specifically, the server performs the following operations:

[1424] Establishing a database connection: Connect to MySQL or MongoDB.

[1425] Inserting Data: Inserting the pre-processed data into the appropriate tables and collections.

[1426] Step 4: Training the AI ​​model

[1427] The server inputs the preprocessed data as training data into the AI ​​model, which then learns to replicate the user's memories and personality. As input, it receives the preprocessed data stored in the database, and as output, it generates a trained AI model. Specifically, the server performs the following operations:

[1428] Model initialization: Initialize a generative AI model such as GPT-4 or BERT.

[1429] Data Batching: Preprocessed data is fed into the model in batches.

[1430] Execute training: Input data into the model and execute training.

[1431] Step 5: Evaluate and save the model

[1432] The server evaluates the AI ​​model after training and checks its performance. As input, it receives the trained AI model and evaluation data, and as output, it generates the evaluation results and a saved model. Specifically, the server performs the following operations:

[1433] Model evaluation: Calculating metrics such as precision and recall.

[1434] Save the model: Save the model with a good evaluation to the file system or cloud storage.

[1435] Step 6: Cloud avatar generation

[1436] The server generates an avatar of the person on the cloud using the model with a good evaluation. It receives the stored AI model as input and generates the avatar on the cloud as output. Specifically, the server performs the following operations:

[1437] Deploying the model: Deploying the AI ​​model to a cloud environment.

[1438] Avatar Generation: Using the deployed model, an avatar is generated that recreates the user's memories and personality.

[1439] (Application example 1)

[1440] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1441] In modern society, there is a demand for providing optimal content for each individual user by utilizing their life log data and characteristic data. However, conventional systems have had difficulty recreating a user's memories and personality and recommending personalized content. Furthermore, there has been a lack of ways for users to maintain communication with acquaintances and family when they are absent, busy, or even after death. A new system is needed to solve these issues.

[1442] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1443] In this invention, the server includes a means for linking life log data from messenger apps, internet search tools, etc. with feature data such as text and voiceprints; a means for AI to recreate the user's memories and personality using the data as training data; a means for generating an avatar of the user on the cloud; and a means for automatically recommending content based on the user's preferences and interests based on the user's life log data and feature data. This makes it possible to recreate the user's memories and personality and provide personalized content. It also allows the user to maintain communication with acquaintances and family even when they are away, busy, or even after death.

[1444] A "messenger app" is an application that allows users to send and receive text messages, voice messages, images, videos, etc.

[1445] An "Internet search tool" is software or a web service that allows users to search for information on the Internet.

[1446] "Life log data" is data recorded about a user's daily life behavior and activities.

[1447] "Characteristic data" is data used to identify individual users, such as their writing or voiceprint.

[1448] "Training data" is a dataset used for AI learning.

[1449] "AI" stands for artificial intelligence, a technology that allows machines to mimic human intelligence.

[1450] "Reproducing the user's memories and personality" means that the AI ​​imitates the user's thinking and behavior patterns based on the user's past actions and statements.

[1451] "Creating a digital avatar on the cloud" means using cloud computing technology to create a digital avatar of a user.

[1452] "Automatic content recommendation" means that AI suggests appropriate information and entertainment based on a user's preferences and interests.

[1453] The system for implementing this invention collects user life log data and characteristic data, recreates the user's memories and personality based on the collected data, and recommends personalized content. Specific embodiments of this system are described below.

[1454] System Configuration

[1455] Hardware

[1456] Server: A server for collecting, processing, and storing data, as well as training and inferencing AI models.

[1457] Device: The device used by the user, such as a smartphone or computer.

[1458] software

[1459] Messenger app: An application that allows users to send and receive text messages, voice messages, images, videos, etc.

[1460] Internet Search Tools: Software or web services that allow users to search for information on the Internet.

[1461] AI model: A generative AI model that recreates a user's memories and personality based on their life log data and feature data, and recommends personalized content.

[1462] Data collection and processing

[1463] The server collects user life log data and feature data from messaging apps and internet search tools, including user message history, search history, and voice data. The collected data is preprocessed and then used as training data for the AI ​​model.

[1464] AI model training and inference

[1465] The server trains the AI ​​model based on the collected data. Specifically, text data is digitized using TfidfVectorizer and a clustering algorithm (e.g., KMeans) is used to create a user profile. Voice data is converted to text using speech recognition technology and processed in the same way.

[1466] Content Recommendations

[1467] The server uses a trained AI model to automatically recommend content based on the user's preferences and interests, including entertainment content such as movies, music, and articles. The recommended content is then sent to the user's device.

[1468] Specific examples

[1469] For example, if User A frequently talks about "movies" on a messaging app, the server collects that data and trains the AI ​​model. If User A's search history frequently includes searches for "latest movie reviews" and "movie trailers," the server creates a profile for User A based on this data. As a result, User A is recommended the latest movie review articles and movie trailer videos.

[1470] Prompt Sentence Examples

[1471] Develop an application that recommends content based on the user's interests and preferences, based on data collected from the user's messaging app and search history. Specifically, implement a feature that analyzes the topics and keywords frequently discussed by the user and automatically recommends content such as movies, music, and articles.

[1472] In this way, a system can be constructed that utilizes users' life log data and characteristic data to deliver personalized content.

[1473] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1474] Step 1:

[1475] The server collects user life log data and feature data from messenger apps and internet search tools, specifically, user message history, search history, voice data, etc. This data is collected using APIs and database queries.

[1476] Input: Raw data from messaging apps and search tools

[1477] Output: Collected lifelog data and feature data

[1478] Step 2:

[1479] The server preprocesses the collected data: text data is digitized using TfidfVectorizer, and voice data is converted to text using speech recognition technology, converting the data into a format suitable for the AI ​​model.

[1480] Input: Collected lifelog data and feature data

[1481] Output: Preprocessed data

[1482] Step 3:

[1483] The server trains the AI ​​model based on the preprocessed data. Specifically, it creates a user profile using a clustering algorithm (e.g., KMeans), which allows the model to reflect the user's behavioral patterns and interests.

[1484] Input: Preprocessed data

[1485] Output: A trained AI model

[1486] Step 4:

[1487] The server uses a trained AI model to automatically recommend content based on the user's preferences and interests, specifically selecting entertainment content such as movies, music, and articles based on the user profile.

[1488] Input: Trained AI model, user profile

[1489] Output: Recommended content

[1490] Step 5:

[1491] The server notifies the user of the recommended content via push notifications or in-app messages.

[1492] Input: Recommended content

[1493] Output: Content displayed on the user's device

[1494] In this way, a system is constructed that utilizes users' life log data and characteristic data to deliver personalized content.

[1495] Example 2

[1496] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1497] In modern society, there is a need to maintain communication with acquaintances, friends, and family even when users are away, busy, or have passed away. However, conventional technologies have struggled to accurately mimic a user's language and thought patterns to achieve natural dialogue. Furthermore, there has been a lack of means to effectively utilize a user's past messages and conversation history to train a generative AI model. This has led to the problem of user avatars generating unnatural responses.

[1498] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1499] In this invention, the server includes: means for linking life log data and feature data such as sentences and voiceprints from messenger apps, internet search tools, etc., means for a generative AI model to reproduce the user's memories and personality using the linked data as training data; means for generating an avatar of the user on the cloud; means for collecting the user's past messages and dialogue history and storing them in a database; means for preprocessing the collected data, cleaning the text, tokenizing, and extracting important phrases and patterns; means for training the generative AI model using the preprocessed data; means for generating an avatar of the user using the trained generative AI model; means for receiving messages from acquaintances and family members, inputting them as prompts into the generative AI model, and generating appropriate replies; and means for sending the generated replies to acquaintances and family members. This enables the system to accurately mimic the user's language and thought patterns and realize natural conversations.

[1500] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[1501] An "Internet search tool" is software that allows users to search for and retrieve information on the Internet.

[1502] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, and health data.

[1503] "Characteristic data" is data that represents a user's individual characteristics, including writing style and voiceprint.

[1504] A "generative AI model" is a model that uses artificial intelligence technology to imitate a user's language usage and thought patterns.

[1505] The "cloud" is a collection of computer resources and services provided over the Internet.

[1506] An "avatar" is a virtual entity that mimics the user's speech and thought patterns and converses on the user's behalf.

[1507] A "database" is a system for efficiently storing, searching, and managing data.

[1508] "Preprocessing" refers to the process of converting data into a format suitable for analysis and model training.

[1509] "Tokenization" is the process of dividing text data into smaller units such as words or phrases.

[1510] A "prompt" is text that is input to a generative AI model and contains instructions for the model to generate an appropriate response.

[1511] This invention is a system for maintaining communication with acquaintances, friends, and family even when the user is away, busy, or has passed away. This system links life log data from messenger apps, internet search tools, etc. with feature data such as text and voiceprints, and uses this data as training data to generate an AI model that reproduces the user's memories and personality. A specific embodiment of this system is described below.

[1512] 1. Collection of User Data

[1513] The server collects the user's past messages and conversation history. This includes emails, chat logs, and social media posts. The server stores this data in a database. For example, the server accesses the user's email account and downloads past emails. It also uses the chat app's API to retrieve the user's chat log.

[1514] 2. Data Preprocessing

[1515] The server pre-processes the collected data, which includes cleaning the text, tokenizing it, and extracting important phrases and patterns. Specifically, the server removes unnecessary HTML tags and special characters from the text, splits the text into words and phrases (tokenization), and extracts frequently occurring phrases and patterns, which are then stored in a database.

[1516] 3. Training the generative AI model

[1517] The server uses the preprocessed data to train a generative AI model. For example, OpenAI's GPT-4 is used here. The server inputs the preprocessed data into the generative AI model and trains the model. Once trained, the model will be able to mimic the user's language and thought patterns.

[1518] 4. The Creation of an Avatar

[1519] The server uses a trained generative AI model to generate an avatar for the user, which mimics the user's speech and thought patterns to replicate how the user normally interacts. The generated avatar is then stored in a database.

[1520] 5. Receiving messages and generating replies

[1521] The device receives messages from acquaintances and family members. The device inputs this message as a prompt into the generative AI model to generate an appropriate reply. For example, if a message is received from an acquaintance saying, "What are your plans for tomorrow?", the device inputs this message into the generative AI model as a prompt. An example of a prompt is "User's name: Yamada Taro, User's past message: 'I'm busy today, so please check my plans for tomorrow,' Prompt: 'When Yamada Taro is busy, please generate a reply to check tomorrow's plans.'"

[1522] 6. Sending a Reply

[1523] The device then sends the generated reply to a friend or family member. For example, it receives a reply from the generative AI model saying, "Tomorrow's schedule is a meeting at 10:00 AM and a client meeting at 2:00 PM," and sends it to a friend or family member.

[1524] In this way, the server and device work together to generate an avatar of the user, enabling them to maintain communication with acquaintances, friends, and family even when they are absent, busy, or have passed away.

[1525] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1526] Step 1: Collect user data

[1527] The server collects the user's past messages and conversation history. Input includes emails, chat logs, and social media posts. The server stores this data in a database. Specifically, the server accesses the email account and downloads past emails. It also uses the chat app's API to retrieve the user's chat log. The collected data is stored in the database as output.

[1528] Step 2: Preprocessing the data

[1529] The server preprocesses the collected data. The input includes the data collected in step 1. The server cleans the text, tokenizes it, and extracts important phrases and patterns. Specifically, the server removes unnecessary HTML tags and special characters from the text, and splits the text into words and phrases (tokenization). It then extracts frequently occurring phrases and patterns and stores them in a database. The output is the preprocessed data.

[1530] Step 3: Training the generative AI model

[1531] The server trains a generative AI model using the preprocessed data. The input includes the data preprocessed in step 2. The server inputs the preprocessed data into the generative AI model and trains the model. Specifically, the server inputs the data into the generative AI model (e.g., GPT-4) and adjusts the model parameters. The output is a fully trained generative AI model.

[1532] Step 4: Creating an Avatar

[1533] The server uses the trained generative AI model to generate an avatar for the user. The input includes the generative AI model trained in step 3. The server inputs the user's characteristics into the generative AI model to generate an avatar. Specifically, the server inputs the user's language and thought patterns into the model to generate an avatar. As an output, the generated avatar is stored in a database.

[1534] Step 5: Receiving the message and generating a reply

[1535] The device receives messages from acquaintances and family members. The input includes messages from acquaintances and family members. The device inputs these messages as prompts to the generative AI model, which then generates an appropriate reply. Specifically, the device uses the messaging app's API to obtain the received message and inputs it as a prompt to the generative AI model. The output is a reply from the generative AI model.

[1536] Step 6: Send your reply

[1537] The device sends the generated reply to the friend or family member. The input includes the reply generated in step 5. Specifically, the device uses the messaging app's API to send the generated reply. The output is the reply sent to the friend or family member.

[1538] In this way, the server and device work together to generate an avatar of the user, enabling them to maintain communication with acquaintances, friends, and family even when they are absent, busy, or have passed away.

[1539] (Application example 2)

[1540] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1541] In modern society, it is often difficult to provide customer service when users are absent or busy. Furthermore, there are limited ways to maintain communication with acquaintances, friends, and family members even after a user passes away. Furthermore, when providing customer service in a virtual store, it is necessary to mimic the user's language and thought patterns, but there is a lack of technology to achieve this.

[1542] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc.; means for AI to reproduce the user's memories and personality using the data as training data; means for generating an avatar of the user on the cloud; means for the user's avatar to interact with customers in a virtual store; means for generating answers to customer questions using a generative AI model; means for imitating the user's language and thought patterns using prompt sentences; and means for installation on a smartphone or head-mounted display. This enables customer interaction even when the user is absent or busy, and allows communication with acquaintances, friends, and family to be maintained even after the user's death. Furthermore, customer interaction in a virtual store that imitates the user's language and thought patterns is possible.

[1543] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[1544] An "Internet search tool" is software that searches for information on the Internet and provides it to users.

[1545] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, health data, etc.

[1546] "Characteristic data" is data used to identify an individual, such as a user's writing or voiceprint.

[1547] "Training data" is data used to train a machine learning model.

[1548] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[1549] The "cloud" is a collection of computer resources and services provided over the Internet.

[1550] An "avatar" is a virtual being created to mimic the user's speech and thought patterns.

[1551] A "virtual store" is a virtual store that exists on the Internet, where users can purchase goods and services.

[1552] "Customer service" refers to responding to and dealing with questions and requests from customers.

[1553] A "generative AI model" is an artificial intelligence model that generates text and speech by imitating a user's language and thought patterns.

[1554] A "prompt" is text that is input to a generative AI model, instructing the model how to respond.

[1555] A "smartphone" is a mobile phone that can connect to the Internet and run applications.

[1556] A "head-mounted display" is a display device worn on the head that provides virtual reality and augmented reality experiences.

[1557] The system for implementing this invention collects user life log data and feature data, and generates an avatar of the user on the cloud based on the collected data. A specific embodiment of this system will be described below.

[1558] First, the server collects the user's life log data and feature data from messenger apps, internet search tools, etc. Life log data includes the user's daily behavior history, location information, health data, etc. Feature data includes data that can be used to identify an individual, such as the user's writing style and voiceprint.

[1559] The server then uses this data as training data to train a generative AI model, which is then used to mimic the user's language and thought patterns. Specifically, it uses prompt sentences to mimic the user's language and thought patterns.

[1560] The server generates an avatar of the user in the cloud. This avatar exists to maintain communication with acquaintances, friends, and family even when the user is absent, busy, or even after the user passes away. The avatar can respond to customers in a virtual store. For example, if a customer asks, "What are the features of this product?", the avatar uses a generative AI model to generate an appropriate answer.

[1561] This system is implemented as an application installed on a smartphone or head-mounted display, and can improve the customer experience by allowing the avatar to respond to customers even when the user is absent or busy.

[1562] As a specific example, by using the following prompt sentence, the avatar can respond to customers by imitating the user's language and thought patterns.

[1563] Example prompt sentence:

[1564] You are a salesperson in a virtual store. Please answer the following questions by imitating the user's language and thought patterns.

[1565] Question: {question}

[1566] In this way, the avatar can respond to customers even when the user is absent or busy, and can maintain communication with acquaintances, friends, and family even after the user's death.It is also possible for the avatar to respond to customers in a virtual store by imitating the user's language and thought patterns.

[1567] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1568] Step 1:

[1569] The server collects users' life log data and feature data from messenger apps and internet search tools. The input includes the user's daily behavior history, location information, health data, text, voiceprint, etc. This data is collected and stored in a database. The output is the collected life log data and feature data.

[1570] Step 2:

[1571] The server preprocesses the collected lifelog data and feature data. The data collected in step 1 is used as input. Data processing such as data cleaning, normalization, and feature extraction is performed, and the data is converted into a format suitable for training a generative AI model. The preprocessed data is obtained as output.

[1572] Step 3:

[1573] The server uses the preprocessed data as training data to train the generative AI model. The preprocessed data from step 2 is used as input. The generative AI model analyzes the data and extracts patterns to learn the user's language and thought patterns. The output is a trained generative AI model.

[1574] Step 4:

[1575] The server generates an avatar for the user on the cloud. The generative AI model trained in step 3 is used as input. Based on the generative AI model, a virtual being is generated that mimics the user's speech and thought patterns. The output is an avatar generated on the cloud.

[1576] Step 5:

[1577] The device runs an application that allows the user's avatar to interact with customers in a virtual store. The input includes the avatar generated on the cloud and the customer's question. The device sends the customer's question to the cloud and generates an appropriate answer using a generative AI model. The output is the answer for the customer.

[1578] Step 6:

[1579] The server uses prompts to mimic the user's language and thought patterns. The input includes the customer's question and the prompt. The generative AI model mimics the user's language and thought patterns based on the prompt and generates an appropriate answer. The output is the mimicked answer.

[1580] Step 7:

[1581] The terminal displays the answer to the customer through an application installed on a smartphone or head-mounted display. It includes the answer generated in step 6 as input. The terminal displays the answer to the customer, improving the customer experience. The output is the answer displayed to the customer.

[1582] Example 3

[1583] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1584] In modern society, there is a demand for digitizing a user's memories and personality and permanently storing and reproducing them. However, conventional systems have difficulty accurately reproducing a user's memories and personality, and lack a mechanism for improving the degree of reproduction as the user uses the system over a long period of time. For this reason, there is a need for a system that can reproduce a user's memories and personality with high accuracy and further improve the accuracy with long-term use.

[1585] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1586] In this invention, the server includes means for linking life log data and feature data such as text and voiceprints from messenger apps, internet search tools, etc., means for AI to reproduce the user's memories and personality using the linked data as training data, means for generating an avatar of the user on the cloud, means for collecting data provided by the user and storing it in a database, means for preprocessing the collected data and converting it into a format suitable for the generative AI model, means for training the generative AI model using the preprocessed data, means for receiving prompts from the user and generating responses using the generative AI model, and means for providing the generated responses to the user. This makes it possible to reproduce the user's memories and personality with high accuracy, and to further improve this accuracy with long-term use.

[1587] A "messenger app" is software that allows users to send and receive text and voice messages.

[1588] An "Internet search tool" is software that allows users to search for information on the Internet.

[1589] "Life log data" refers to data related to a user's daily life, including behavioral history and activity records.

[1590] "Characteristic data" is data that indicates the individual characteristics of a user, and includes sentences, voiceprints, etc.

[1591] "Training data" is data used to train a generative AI model, and is a dataset that has been assigned a correct answer label.

[1592] A "generative AI model" is an artificial intelligence model used to recreate a user's memories and personality.

[1593] The "cloud" is a collection of computer resources and services provided over the Internet.

[1594] An "avatar" is a digital being that recreates the user's memories and personality.

[1595] A "database" is a system for systematically storing and managing collected data.

[1596] "Preprocessing" refers to the process of converting collected data into a format suitable for a generative AI model.

[1597] A "prompt" is a question or request that a user enters into a system.

[1598] A "response" is the answer that a generative AI model generates in response to a prompt.

[1599] This invention is a system that digitizes a user's memories and personality and generates an avatar on the cloud. The system collects life log data and feature data from messenger apps, internet search tools, etc., and uses this data as training data for a generative AI model. A specific embodiment of this system is described below.

[1600] First, users provide life log data such as text messages, voice data, and image data through messenger apps or internet search tools. The device then sends this data to a server, which then stores the received data in a database.

[1601] The server then preprocesses the collected data. Specifically, text data is tokenized using natural language processing (NLP) techniques, audio data is converted to text using speech recognition software, and image data is subjected to feature extraction using image recognition techniques. Machine learning frameworks such as TensorFlow and PyTorch are used for this preprocessing.

[1602] The preprocessed data is used to train the generative AI model. The server uses this data to train the generative AI model, teaching it patterns to reproduce the user's memories and personality. As the learning progresses, the model's reproducibility improves.

[1603] When a user inputs a question or request into the system as a prompt, the device sends the prompt to the server, which uses a generative AI model to generate an appropriate response to the prompt, which is then sent to the device and provided to the user.

[1604] For example, if a user asks, "What is my favorite movie?", the server uses a generative AI model to search for information about "favorite movies" from the user's past data and generate a response such as, "Your favorite movie is Inception." Similarly, if a user asks, "Where was the last place I traveled?", the server generates a response such as, "You went to Hawaii last summer."

[1605] This system employs a monthly subscription system, and the longer a user continues to pay, the more data is collected and the more the AI ​​learns. This improves the reproducibility of the avatar, making it possible to more accurately reproduce the user's memories and personality. The flow of the identification process in Example 3 will be explained using Figure 15.

[1606] Step 1:

[1607] Users provide lifelog data such as text messages, voice data, and image data through messenger apps and internet search tools. The device sends this data to the server. The input is the lifelog data from the user, and the output is the data sent to the server.

[1608] Step 2:

[1609] The server stores the received data in a database. Specifically, it stores text messages, voice data, and image data in the corresponding database tables. The input is the data sent from the terminal, and the output is the data stored in the database.

[1610] Step 3:

[1611] The server preprocesses the collected data. Text data is tokenized using natural language processing (NLP) techniques to extract important keywords. Voice data is converted to text using voice recognition software. Image data is subjected to feature extraction using image recognition techniques. The input is raw data stored in a database, and the output is preprocessed data.

[1612] Step 4:

[1613] The server uses the preprocessed data to train a generative AI model. Specifically, it uses machine learning frameworks such as TensorFlow and PyTorch to train the model to learn patterns that will reproduce the user's memories and personality. The input is the preprocessed data, and the output is a trained generative AI model.

[1614] Step 5:

[1615] The user inputs a question or request to the system as a prompt. The terminal sends this prompt to the server. The input is the prompt from the user, and the output is the prompt sent to the server.

[1616] Step 6:

[1617] The server inputs the received prompt into the generative AI model to generate an appropriate response. The generative AI model generates the most appropriate answer based on the user's past data. The input is the prompt and the trained generative AI model, and the output is the generated response.

[1618] Step 7:

[1619] The server sends the generated response to the terminal, which displays this response to the user. The input is the generated response, and the output is the response provided to the user.

[1620] Specifically, when a user asks, "What's my favorite movie?", the server uses the generative AI model to search for information about "favorite movies" from the user's past data and generates a response such as, "Your favorite movie is Inception." Similarly, when a user asks, "Where was the last place I traveled?", the server generates a response such as, "You went to Hawaii last summer."

[1621] (Application example 3)

[1622] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1623] In conventional systems, the accuracy of the avatars that reproduce the user's memories and personality was low, making it difficult to recommend individually customized content. Furthermore, there was a lack of a mechanism for improving the accuracy of the system as the user uses it over a long period of time, making it difficult to improve the user experience.

[1624] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for linking life log data and feature data such as sentences and voiceprints from messenger apps, internet search tools, etc., a means for AI to reproduce the user's memories and personality using the data as training data, a means for generating an avatar of the user on the cloud, and a means for collecting user data, generating an avatar using an AI model, and recommending individually customized content. This makes it possible to reproduce the user's memories and personality with high accuracy and recommend individually customized content.

[1625] A "messenger app" is software that allows users to send and receive text messages, voice messages, images, videos, etc.

[1626] An "Internet search tool" is software that allows users to search for information on the Internet.

[1627] "Life log data" refers to data related to a user's daily life, including behavioral history, location information, health data, etc.

[1628] "Characteristic data" is data used to identify an individual, such as a user's writing or voiceprint.

[1629] "Training data" is data used to train a machine learning model.

[1630] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence.

[1631] "Recreating the user's memories and personality" means imitating the user's past experiences and personality based on collected data.

[1632] The "cloud" is a collection of computer resources and services provided over the Internet.

[1633] An "avatar" is a virtual being that recreates the user's memories and personality.

[1634] "Personalized Content" is information and entertainment specially selected based on a user's preferences and interests.

[1635] An "AI model" is a mathematical model that uses artificial intelligence algorithms to analyze data and make predictions and classifications.

[1636] A "monthly subscription" is a pricing system in which users can use a service by paying a fixed monthly fee.

[1637] A system for implementing this invention is configured as follows: The server has a means for linking life log data with feature data such as text and voiceprints from messenger apps, internet search tools, etc. This makes it possible to collect data about the user's daily life and data for identifying individuals.

[1638] Next, the server has a means for the AI ​​to recreate the user's memories and personality using the collected data as training data. Specifically, based on the collected life log data and feature data, the AI ​​model mimics the user's past experiences and personality. This AI model is implemented using OpenAI's API.

[1639] Furthermore, the server has a means for generating an avatar of the user on the cloud. The generated avatar is a virtual entity that reproduces the user's memories and personality, and is managed on the cloud.

[1640] The server also has a means for collecting user data, generating an avatar using an AI model, and recommending individually customized content. Specifically, the server collects data such as the user's viewing history, preferences, and feedback, and uses the AI ​​model to recommend the most suitable content to the user based on this data. This recommendation provides information and entertainment specially selected based on the user's preferences and interests.

[1641] As users continue to pay over time, the system collects more data and the AI ​​learns more, allowing the avatar to become more realistic and more accurately recreate the user's memories and personality.

[1642] As a concrete example, consider a case where a user likes "action movies" and "comedy movies," likes "movie1," and dislikes "movie2." An example of a prompt sentence that generates an avatar based on this data and recommends content is as follows:

[1643] Example prompt sentence:

[1644] Generate an avatar that recreates the user's memories and personality based on the user's data: {"user_id": "user123", "viewing_history": ["movie1", "movie2"], "preferences": ["action", "comedy"], "feedback": ["liked movie1", "disliked movie2"]}.

[1645] By inputting this prompt into OpenAI's API, an avatar that replicates the user's memories and personality is generated, and the system can then recommend the most appropriate content to the user.

[1646] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1647] Step 1:

[1648] The server collects life log data and feature data from messenger apps and internet search tools. Specifically, it obtains users' text messages, voice messages, search history, location information, etc. The input is various user data, and the output is the collected life log data and feature data.

[1649] Step 2:

[1650] The server inputs the collected life log data and feature data into the AI ​​model as training data. Specifically, it preprocesses this data and converts it into a format that the AI ​​model can learn from. The input is the preprocessed data, and the output is the training data used to train the AI ​​model.

[1651] Step 3:

[1652] The server uses an AI model to generate an avatar that replicates the user's memories and personality. Specifically, it uses OpenAI's API to generate prompts based on the user's data and input them into the AI ​​model. The input is the prompt, and the output is the generated avatar.

[1653] Step 4:

[1654] The server stores the generated avatar on the cloud. Specifically, it uses a cloud storage service to securely store the avatar data. The input is the generated avatar, and the output is the avatar data stored on the cloud.

[1655] Step 5:

[1656] The server collects data such as the user's viewing history, preferences, and feedback, and uses an AI model to recommend individually customized content. Specifically, it generates prompts based on the user's data and inputs them into the AI ​​model. The input is the user's viewing history and preferences, and the output is the recommended content.

[1657] Step 6:

[1658] The user receives the recommended content and views or uses it. Specifically, the user displays and enjoys the recommended content using a smartphone or other device. The input is the recommended content, and the output is the user's viewing or usage history.

[1659] Step 7:

[1660] As users continue to pay over time, the server collects more data and continues to train the AI ​​model. Specifically, the server periodically collects user usage data through a monthly subscription and retrains the AI ​​model. The input is continuously collected user data, and the output is the retrained AI model.

[1661] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1662] "Example 1"

[1663] One embodiment of the present invention is a system incorporating an emotion engine. This system recognizes the user's emotions and adjusts the avatar's response accordingly. Specifically, when the user is happy, the avatar will speak in a bright tone. Conversely, when the user is sad, the avatar will speak in a calm tone. In this way, the emotion engine grasps the user's emotional state in real time and adjusts the avatar's response accordingly.

[1664] "Example 2"

[1665] The emotion engine also controls the avatar's behavior according to the user's emotional state. For example, when the user is angry, the avatar chooses to apologize. When the user is happy, the avatar chooses to congratulate. In this way, the emotion engine understands the user's emotional state and controls the avatar's behavior accordingly.

[1666] "Example 3"

[1667] Furthermore, the emotion engine controls the facial expression of the avatar according to the user's emotional state. For example, when the user is happy, the avatar makes a smiling face. When the user is sad, the avatar makes a sad face. In this way, the emotion engine understands the user's emotional state and controls the facial expression of the avatar accordingly.

[1668] The processing flow of each embodiment will be described below.

[1669] "Example 1"

[1670] Step 1: The emotion engine understands the user's emotional state in real time.

[1671] Step 2: The emotion engine adjusts the avatar's response according to the emotional state it has determined.

[1672] Step 3: The avatar responds with a tailored response to the user.

[1673] "Example 2"

[1674] Step 1: The emotion engine understands the user's emotional state in real time.

[1675] Step 2: The emotion engine controls the avatar's behavior according to the emotional state it has grasped.

[1676] Step 3: The avatar performs the controlled action on the user.

[1677] "Example 3"

[1678] Step 1: The emotion engine understands the user's emotional state in real time.

[1679] Step 2: The emotion engine controls the avatar's facial expression according to the emotional state it has grasped.

[1680] Step 3: The avatar displays a controlled facial expression to the user.

[1681] Example 1

[1682] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1683] In modern society, there is a demand f...

Claims

1. A means for acquiring feature data representing a person's life log data, a text about said person, or a voiceprint of said person; A means for inputting a message input by a user to a generation AI that has been trained based on the feature data, and outputting data output from the generation AI; A system including:

2. The person is a deceased person, The system of claim 1 .

3. When the generation AI is operated, a cost according to the operation time of the generation AI is added, and the higher the cost, the higher the reproduction degree of the person's memory and personality is. A system according to claim 1 or claim 2.

Images