System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to upload personal data associated with a deceased person; means for receiving and storing the uploaded personal data; means for extracting features from the stored personal data; means for training a AI model based on the extracted features; and means for providing an interface through which the user can communicate with the deceased person using the trained AI model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, there are limited means to pass on personal knowledge, skills, conversational styles, and so on to future generations. In particular, there is a lack of means to pass on the skills and knowledge of the deceased, and the lack of successors due to the declining birthrate and aging population is becoming a serious problem. As this puts the risk of valuable knowledge and culture being lost, there is a need for a system that can reproduce the characteristics and patterns of the deceased and pass them on to the next generation. [Means for solving the problem]

[0005] The present invention provides a system in which users upload personal data related to the deceased, and a server receives and stores that data. The stored personal data is preprocessed and converted into a unified format to extract features while maintaining data consistency. An AI model is trained based on the extracted features, and an interface is provided that allows users to communicate with the deceased using the trained AI model. The system also includes a means for converting voice data to text, making both text and voice data available. This allows the deceased's knowledge and skills to be realistically reproduced and passed on to the next generation.

[0006] "User" means an entity that uses this system to provide personal data relating to a deceased person.

[0007] A "deceased person" is a person who has already passed away and whose related data is reproduced through the system.

[0008] "Personal data" is information used to extract characteristics and patterns of the deceased, such as text data or audio data about the deceased.

[0009] "Uploading" is the act of a user sending data from their computer or device to a server.

[0010] "Receiving" is the act of the server receiving data sent by the user.

[0011] "Storage" refers to the act of storing and preserving received data in a database or cloud storage.

[0012] "Preprocessing" refers to the act of preparing the stored data in a form suitable for subsequent processing, such as by deleting unnecessary parts from the data and converting it into a unified format.

[0013] "Features" are unique patterns or elements extracted from the data that represent the deceased's conversational style, knowledge, and skills.

[0014] "Extraction" is the act of extracting features from stored data.

[0015] An "AI model" is a machine learning algorithm that reproduces the characteristics and patterns of the deceased by learning their features.

[0016] "Learning" is the process by which the AI ​​model understands the characteristics and patterns of the deceased based on the data provided.

[0017] An "interface" is a means by which a user can directly interact with an AI model and communicate with the recreated features and patterns of the deceased person.

[0018] "Audio data" means a digital audio file containing a recording of the deceased person's utterances or speech.

[0019] "Convert to text" refers to the act of converting audio data into text format. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system that can reproduce the characteristics and patterns of a deceased person and communicate with them. This system is realized mainly by training an AI model on a server based on personal data provided by the user.

[0042] User Actions

[0043] Using a dedicated application or web interface, users upload personal data (e.g., text data, audio data) related to the deceased, such as diary entries, blogs, message histories, or audio recordings written by the deceased.

[0044] Receiving and storing data

[0045] The server receives the personal data uploaded by the user, which is then encrypted for security purposes and stored in a database or cloud storage, ensuring the confidentiality of the data.

[0046] Data Preprocessing

[0047] The server performs preprocessing on the stored data. This preprocessing includes removing unnecessary symbols and tags from text data, and converting audio data into text using a speech recognition API. This converts the data into a unified format, allowing for smooth subsequent processing.

[0048] Feature extraction

[0049] After preprocessing, the server extracts features from the data. Specifically, it uses natural language processing technology to analyze keywords, distinctive phrases, and grammatical structures in the text to identify the deceased's writing style and speaking patterns. In the case of audio data, the server also extracts speech rhythm, intonation, and specific pronunciation patterns.

[0050] Model training

[0051] After the features are extracted, the server trains an AI model based on the extracted features. For example, a Transformer-based model is built and trained using a machine learning library. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge. This trained AI model is then used to interact with the user.

[0052] Providing an interface

[0053] Once the trained AI model is complete, the server integrates it into a user-accessible interface (e.g., a chatbot or voice response system). This interface allows users to easily access and enjoy virtual conversations with the deceased. A dedicated chat application or web interface is provided, through which users can ask questions and enjoy conversations with the AI.

[0054] Specific examples

[0055] 1. User data provision:

[0056] The user opens the application, selects and uploads the diary file of the deceased person.

[0057] 2. Receiving and storing data:

[0058] The server receives the diary files of the deceased person, encrypts the data, and stores it in cloud storage.

[0059] 3. Data preprocessing:

[0060] The server cleans the diary text and removes unnecessary HTML tags.

[0061] 4. Feature extraction:

[0062] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[0063] 5. Train the model:

[0064] The server uses a machine learning library to train a Transformer model based on the features.

[0065] 6. Providing an interface:

[0066] The server integrates the trained model into the application and makes the chatbot available for users to access.

[0067] 7. Communication:

[0068] Through the application, the user types, "What's the weather like today?" and the AI ​​responds, "It's sunny today," in the style of the deceased.

[0069] In this way, the system can accurately reproduce the characteristics and conversational style of the deceased and engage in dialogue with the user, thereby enabling the knowledge and skills of the deceased to be passed on to future generations, contributing to the preservation of culture and the continuation of industry.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user selects and uploads personal data related to the deceased (e.g., text files, audio files) through a dedicated application or web interface.

[0073] Step 2:

[0074] The terminal transmits the data selected by the user to the server, and before transmitting the data, it is encrypted as necessary.

[0075] Step 3:

[0076] The server receives the data sent from the device, and immediately after receiving it, it re-encrypts the data and stores it securely in a database or cloud storage.

[0077] Step 4:

[0078] The server starts preprocessing the stored data. In the case of text data, unnecessary spaces and special characters are removed and the format is standardized.

[0079] Step 5:

[0080] If voice data is included, the server converts the voice data into text data using a voice recognition API, and this converted text data also undergoes preprocessing.

[0081] Step 6:

[0082] After preprocessing is complete, the server uses natural language processing technology to extract features from the data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the characteristics of the deceased.

[0083] Step 7:

[0084] The server uses the extracted features to train an AI model using a machine learning library, for example, by building and training a Transformer-based model.

[0085] Step 8:

[0086] The trained AI model is tested and evaluated, and adjustments are made as needed. The server manages the versioning of the generated model and stores it securely.

[0087] Step 9:

[0088] The server then integrates the trained AI model into a user interface (chatbot or voice response system), allowing users to interact with the recreated features of the deceased.

[0089] Step 10:

[0090] Users enter questions or messages through the interface, which are sent to the AI ​​model on the server, which uses the model to generate an appropriate response and sends the answer back to the user.

[0091] Step 11:

[0092] The terminal displays the response received from the server to the user, and the user can continue the conversation by sending another question or message.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] It is necessary to provide a system that can reproduce the characteristics and communication style of the deceased and converse with them while ensuring the security of personal data. It is also difficult to deal with various forms of personal data, including voice data, and to effectively train an AI model to realize the dialogue.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes means for a user to upload personal data related to the deceased, means for receiving and storing the uploaded personal data, means for encrypting the personal data and storing it in cloud storage or a database, means for preprocessing the stored personal data and converting it into a unified format, means for extracting features from the preprocessed personal data, means for training an AI model based on the extracted features, means for providing an interface using the trained AI model to allow the user to communicate with the deceased, and means for responding to the user's questions and messages through the provided interface. This makes it possible to extract features from various forms of data about the deceased and effectively train the AI ​​model to realize dialogue while ensuring the security of the personal data.

[0098] "Uploading" is the act of a user sending data from their device to a server.

[0099] "Personal data" refers to all information generated by an individual, including text data and audio data relating to a deceased person.

[0100] "Encryption" is the process of converting data using a specific algorithm to make it unreadable to third parties.

[0101] "Cloud storage" is an external storage service that stores and accesses data over the Internet.

[0102] "Preprocessing" is the process of cleaning and formatting data to make it easier for subsequent processing.

[0103] "Features" are important elements or parameters extracted from data, and are information used to train AI models.

[0104] An "AI model" is a learning algorithmic structure built using artificial intelligence techniques and trained to perform a specific task.

[0105] "Interface" refers to the tools and environments through which users interact with a system, including chatbots and voice response systems.

[0106] "Voice recognition technology" is a technology that analyzes voice data and converts it into text.

[0107] A "Transformer model" is a type of deep learning model used in natural language processing, capable of analyzing the meaning and context of text.

[0108] The present invention is a system that can reproduce the characteristics and patterns of a deceased person and communicate with the deceased. To implement this system, the entire system is constructed based on the following procedure.

[0109] First, users use a dedicated application or web interface to upload personal data related to the deceased, including diary entries, blogs, message history, or audio recordings.

[0110] The server then receives the personal data uploaded by the user, encrypts the data, and stores it in cloud storage or a database. The received data is encrypted using the Advanced Encryption Standard (AES) to ensure security. The encryption key is stored in a separate, secure location.

[0111] The server preprocesses the stored data. For text data, unnecessary symbols and HTML tags are removed and the data is normalized. For audio data, the data is converted to text using Google Cloud Speech-to-Text or other speech recognition APIs.

[0112] After preprocessing, the server extracts features from the data. For example, natural language processing techniques such as TF-IDF (Term Frequency-Inverse Document Frequency) and Word2Vec are used to extract important keywords and unique phrases from the text. In the case of audio data, features such as speech rhythm, intonation, and specific pronunciation patterns are also extracted.

[0113] After the features are extracted, the server trains an AI model based on them. It uses machine learning libraries such as TensorFlow and PyTorch to build a Transformer-based model, using the extracted features as training data. Through an iterative process, it learns the deceased person's unique speaking style and knowledge.

[0114] Using the trained AI model, the server provides a user-accessible interface, consisting of a chatbot and a voice response system, allowing users to enjoy virtual conversations with the deceased through a dedicated chat application or web interface.

[0115] As a specific example of operation, a user opens the application, selects and uploads the diary file of the deceased person. The server then encrypts, stores, and preprocesses the file, extracts features, and trains the AI ​​model. Finally, when the user asks through the application, "What's the weather like today?", the AI ​​responds, "It's sunny today," in the style of the deceased person.

[0116] This system can accurately reproduce the characteristics and conversational style of the deceased and enable dialogue with the user, thereby passing on the knowledge and skills of the deceased to the next generation and contributing to the preservation of culture and the continuation of industry.

[0117] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0118] Step 1: Upload user data

[0119] The user opens a dedicated application or web interface, selects a data file (e.g., diary, audio file) related to the deceased, and clicks the upload button. The input data is the personal data selected by the user, and the output is to send the selected file to the server. Specifically, the user clicks the file selection button in the application, selects "diary.txt," and presses the upload button.

[0120] Step 2: Receiving and Encrypting Data

[0121] The server stores the received personal data in a temporary folder. It then encrypts the received data using the AES (Advanced Encryption Standard) algorithm. The input data is the raw personal data sent by the user, and the output is the encrypted data. Specifically, the server receives "diary.txt," encrypts it using the AES algorithm, and saves it in an AWS S3 bucket as "encrypted_diary.txt."

[0122] Step 3: Preprocessing the data

[0123] The server loads the stored data and begins preprocessing. The input data is encrypted personal data, and the output is cleaned and normalized data. The processing involves removing unnecessary symbols and HTML tags from text data, and converting audio data to text using Google Cloud Speech-to-Text. Specifically, it loads "encrypted_diary.txt," decrypts it, and then removes HTML tags and special characters.

[0124] Step 4: Feature extraction

[0125] The server extracts features from the data after preprocessing. The input data is cleaned personal data, and the output is the extracted features. Specific techniques include using natural language processing techniques (e.g., TF-IDF, Word2Vec) to extract important keywords and distinctive phrases from the text. In the case of audio data, features of audio patterns are extracted. Specific operations include extracting feature keywords from "diary_clean.txt" using TF-IDF and saving them in "keyword_list.txt."

[0126] Step 5: Training the AI ​​model

[0127] The server builds an AI model based on the features and trains it. The input data are the extracted features, and the output is a trained AI model. Specifically, a Transformer-based model is created using TensorFlow or PyTorch, and specific features are used as training data. Specifically, a Transformer model is built using TensorFlow, and the extracted "keyword_list.txt" is used as training data to train the model.

[0128] Step 6: Providing an Interface

[0129] The server uses the trained AI model to provide an interface that users can access. The input data is input from the user (text or voice), and the output is the response from the AI ​​model. Specifically, the trained model is integrated into a Flask web application to provide a chat interface. Users can interact with the AI ​​through a dedicated chat application or web interface.

[0130] Step 7: Communicate with users

[0131] The user inputs questions or messages through the interface, and the server responds using a trained AI model. The input data is the user's question or message, and the output is the response from the AI ​​model. Specifically, when a user types "What's the weather like today?" into a chat application, the server responds "It's sunny today" in the style of the deceased.

[0132] (Application example 1)

[0133] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] Currently, there are limited ways to reminisce about memories with deceased family and friends. Traditional photo albums and videos are merely static records and do not allow for interactive communication with the deceased. While there are technologies that utilize personal data to recreate the characteristics and speaking style of the deceased, there is a lack of effective integration of advanced natural language processing and voice reproduction technologies.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0136] In this invention, the server includes: means for a user to upload personal data related to the deceased; means for receiving and storing the uploaded personal data; means for extracting features from the stored personal data; means for training a generative AI model based on the extracted features; means for providing an interface that allows the user to communicate with the deceased using the trained AI model; means for reproducing the voice and conversation style of the deceased based on the responses of the generative AI model; means for inputting prompts that the user can use to enjoy a conversation with the deceased; and means for performing preprocessing to convert voice data into text data. This enables two-way virtual communication with the deceased, allowing the user to reminisce about their memories in a more realistic and moving way.

[0137] "Means by which users can upload personal data relating to a deceased person" refers to devices or software that allow text and audio data relating to a deceased person to be sent digitally to a server.

[0138] "Means for receiving and storing uploaded personal data" refers to the technology or system that receives submitted personal data and stores it in a secure location.

[0139] "Means for extracting features from stored personal data" refers to algorithms or processes that analyze stored data and identify the characteristics and speech patterns of the deceased.

[0140] "Means for training a generative AI model" refers to the process of feeding data to an AI model using the extracted features to train the model to have the ability to reproduce the speech pattern and characteristics of the deceased.

[0141] "Means for providing an interface" means a user interface that allows a user to interact with an AI model, such as a chat application or a voice response system.

[0142] "Means of recreating the voice and speaking style of the deceased based on the response of a generative AI model" refers to technologies and systems that synthesize the voice and distinctive speaking style of the deceased based on the output of an AI model.

[0143] "Means for inputting prompt text" refers to a device or interface that allows a user to input text to ask or speak to the deceased.

[0144] The "means for performing preprocessing to convert voice data into text data" refers to a technology for generating text from a voice file, such as a voice recognition system.

[0145] This invention is a system that allows users to upload personal data related to the deceased, enabling communication that recreates the characteristics and patterns of the deceased.

[0146] 1. Program Generation

[0147] The user uses their device to upload text and audio data related to the deceased to the server. The server receives this data and stores it in a secure location. The server also extracts features from the stored data and analyzes the deceased's unique speaking style and expressions. Based on these features, the server trains a generative AI model to build a model that mimics the characteristics of the deceased. The server then uses the trained AI model to provide an interface that allows the user to interact with the deceased.

[0148] 2. Explanation of program processing

[0149] This system uses the following major hardware and software:

[0150] Server: Used for data storage and computation. A good example is Amazon EC2.

[0151] Generative AI model: Uses OpenAI API to recreate the speaking style of the deceased.

[0152] Web framework: Django is used to provide the user interface.

[0153] Data processing and calculation:

[0154] The server receives and stores the uploaded personal data. It then preprocesses the stored data and converts it into a unified format. For example, it uses a speech-to-text API to convert voice data into text data. It then uses natural language processing technology to extract features and trains a generative AI model. Once trained, the AI ​​model can reproduce the deceased person's unique speaking style.

[0155] 3. Specific Examples

[0156] Below are some specific scenarios:

[0157] 1. The user opens the application and uploads an audio file of the deceased person.

[0158] 2. The server receives the audio file and stores it in secure cloud storage.

[0159] 3. Preprocess the audio file and convert it to text, for example, using the Google Speech-to-Text API.

[0160] 4. The server analyzes the text data and extracts keywords and phrases characteristic of the deceased.

[0161] 5. Based on the extracted features, a generative AI model is trained using the OpenAI API.

[0162] 6. Using the learning model, users can enjoy interacting with their deceased loved ones through chat applications.

[0163] 7. When the user types "Hello, Grandpa," the system responds by recreating the deceased's distinctive speaking style, "Hello, how are you?"

[0164] Prompt Sentence Examples

[0165] "Grandpa, how was your day today?"

[0166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0167] Step 1: The user uses a device to upload text and audio data related to the deceased person to the server.

[0168] Input: Text data and audio data

[0169] Specific operation: The user uses the application's upload function to select and send a note or audio file of the deceased person, which is then received by the server.

[0170] Step 2: The server receives the uploaded personal data and stores it in a secure location.

[0171] Input: Text and audio data uploaded by users

[0172] Output: Stored personal data

[0173] What it does: Encrypts received data and stores it in cloud storage. Uses security protocols to ensure data confidentiality.

[0174] Step 3: The server preprocesses the stored data and converts it into a unified format.

[0175] Input: Stored personal data

[0176] Output: Formatted text data

[0177] Specific operations: For text data, remove unnecessary tags and symbols. For audio data, convert audio to text using the Speech-to-Text API.

[0178] Step 4: The server extracts features from the preprocessed data.

[0179] Input: Formatted text data

[0180] Output: Extracted features

[0181] What it does: It uses natural language processing technology to analyze keywords and distinctive phrases in texts to identify patterns in the speech and writing style of the deceased.

[0182] Step 5: The server trains a generative AI model based on the extracted features.

[0183] Input: Extracted features

[0184] Output: A trained generative AI model

[0185] What it does: It uses a machine learning library (e.g., Transformer) to train an AI model, which iterates to learn the deceased person's speaking style.

[0186] Step 6: The server uses the trained AI model to provide an interface that allows the user to interact with the deceased.

[0187] Input: A trained generative AI model

[0188] Output: User interface

[0189] What it does: Build an interface that acts as a chatbot or voice response system and provides it to users in an easily accessible way.

[0190] Step 7: The user uses the terminal to enter a prompt and interact with the deceased.

[0191] Input: prompt statement

[0192] Output: Response text or audio

[0193] How it works: When a user enters a prompt phrase, such as "Hello, Grandpa," the server uses a generative AI model to generate a response in the style of the deceased person, and returns it as text or voice.

[0194] Through these steps, the system can accurately reproduce the characteristics and conversation style of the deceased person and engage in a virtual dialogue with the user.

[0195] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0196] The present invention includes a system that reproduces the characteristics and patterns of the deceased and enables communication between the deceased and the user, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. This system is realized mainly through the process of providing personal data by the user, data processing by the server, learning of an AI model, and emotion recognition and response generation.

[0197] User operations

[0198] Using a dedicated application or web interface, users select and upload personal data (e.g., text files, audio files) related to the deceased, such as diary entries, blogs, message histories, or audio recordings written by the deceased.

[0199] Receiving and storing data

[0200] The server receives the personal data uploaded by the user, encrypts the data immediately after receiving it, and stores it securely in a database or cloud storage, ensuring the confidentiality and integrity of the data.

[0201] Data Preprocessing

[0202] The server performs preprocessing on the stored data. This preprocessing includes removing unnecessary spaces and symbols in the case of text data, and converting audio data into text using a speech recognition API. This converts the data into a unified format, allowing for smooth subsequent processing.

[0203] Feature extraction

[0204] After preprocessing is complete, the server uses natural language processing technology to extract features from the data. Specifically, it analyzes keywords, phrases, and grammatical structures in the text to identify the deceased's writing style and speaking patterns. In the case of audio data, it also extracts the rhythm and intonation of speech, as well as specific pronunciation patterns.

[0205] Model training

[0206] After the features are extracted, the server trains an AI model based on the extracted features. For example, a Transformer-based model is built and trained using a machine learning library. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge. This trained AI model is then used to interact with the user.

[0207] Emotion Recognition and Response Generation

[0208] The emotion engine recognizes the user's emotions in response to input from the user. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the AI ​​model and reflected when generating responses. This allows for the generation of appropriate responses according to the user's emotional state, resulting in more human-like interactions.

[0209] Providing an interface

[0210] The server provides users with an interface (e.g., a chatbot or voice response system) that integrates the trained AI model and emotion engine. Through this interface, users can enjoy conversations based on the recreated features and emotion recognition of the deceased person.

[0211] Specific examples

[0212] 1. User Data Provision:

[0213] The user opens the application, selects and uploads the diary and audio files of the deceased person.

[0214] 2. Receiving and storing data:

[0215] The server receives the deceased person's diary files and audio files, encrypts the data, and stores it in cloud storage.

[0216] 3. Data preprocessing:

[0217] The server cleans the diary text, removing unnecessary spaces and symbols, and converts the audio file into text using a speech recognition API.

[0218] 4. Feature extraction:

[0219] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[0220] 5. Train the model:

[0221] The server uses a machine learning library to train a Transformer model based on the features.

[0222] 6. Emotion Recognition and Response Generation:

[0223] In response to text or voice input from the user, the server uses an emotion engine to identify the user's emotion, and the AI ​​model generates the optimal response based on that emotion.

[0224] 7. Providing an interface:

[0225] The server provides a chatbot that integrates a trained model and an emotion engine, making it accessible to users.

[0226] 8. Communication:

[0227] The user types, "How are you feeling today?" and the server's AI model recognizes the user's emotion of joy and replies, "I feel great today," in the style of the deceased.

[0228] In this way, the system can not only accurately reproduce the characteristics and conversational style of the deceased, but also identify the user's emotions and provide appropriate responses based on those emotions, enabling richer communication.

[0229] The processing flow will be explained below.

[0230] Step 1:

[0231] The user selects and uploads personal data related to the deceased (e.g., text files, audio files) through a dedicated application or web interface.

[0232] Step 2:

[0233] The terminal transmits the data selected by the user to the server, and before transmitting the data, it is encrypted as necessary.

[0234] Step 3:

[0235] The server receives the data sent from the device, then immediately encrypts it again and stores it securely in a database or cloud storage.

[0236] Step 4:

[0237] The server starts preprocessing the stored data. For text data, unnecessary spaces and special characters are deleted and the format is standardized. For audio data, a speech recognition API is used to convert it into text data.

[0238] Step 5:

[0239] The server uses natural language processing technology to extract features from the preprocessed data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the characteristics of the deceased. In the case of audio data, it also extracts the rhythm and intonation of speech, as well as specific pronunciation patterns.

[0240] Step 6:

[0241] The server uses the extracted features to train an AI model using a machine learning library, such as building and training a Transformer-based model. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge.

[0242] Step 7:

[0243] The server uses an emotion engine to identify the emotion of user input (text and voice). The emotion engine performs text and voice analysis to determine emotional states such as joy, sadness, and anger.

[0244] Step 8:

[0245] The server adjusts the AI ​​model's response based on the emotional information obtained from the emotion engine. By incorporating the emotional information into response generation, an appropriate response is generated according to the user's emotional state.

[0246] Step 9:

[0247] The server provides an interface that integrates trained AI models and emotion engines. For example, users can interact with AI through chatbots or voice response systems.

[0248] Step 10:

[0249] When a user enters a question or message through the interface, it is sent to the server's AI model and emotion engine. The server uses the emotion engine to identify the user's emotion and uses the AI ​​model to generate an appropriate response. This response is then sent back to the user.

[0250] Step 11:

[0251] The device displays the response received from the server to the user. The user can also send questions or messages again, allowing for a continuous dialogue. This allows the user to experience a more human-like dialogue through an AI model that recognizes emotions and recreates the characteristics of the deceased.

[0252] Example 2

[0253] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0254] A system that enables conversation with the deceased must accurately reproduce the characteristics and conversational style of the deceased, recognize the user's emotions, and reflect them in the conversation. It must also convert various data formats into a unified format and efficiently extract features while ensuring the confidentiality and integrity of personal data.

[0255] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0256] In this invention, the server includes: a means for a user to upload personal data related to the deceased; a means for receiving and saving the uploaded personal data; a means for preprocessing the saved personal data, such as deleting unnecessary spaces and symbols in the case of text data and converting audio data into text using speech recognition technology; a means for extracting features from the preprocessed data; a means for training an AI model based on the extracted features to reproduce the deceased's unique conversational style; a means for recognizing emotions in response to user input and generating appropriate responses according to those emotions; and a means for providing an interface that allows the user to communicate with the deceased using the trained AI model and emotion engine. This makes it possible to reproduce the deceased's characteristics and conversational style while generating responses according to the user's emotions.

[0257] A "user" is a person who wishes to use the system to interact with the deceased.

[0258] A "deceased person" is someone who has already passed away, and the system will recreate that person's characteristics and conversation style.

[0259] "Personal data" is information related to a deceased person that is uploaded by a user, including text files, audio files, diaries, message history, etc.

[0260] "Means for uploading" refers to the method by which a user provides personal data relating to the deceased person to the system, including a dedicated application or web interface.

[0261] "Means of receiving and storing" refers to the process by which the server receives the uploaded personal data and stores it safely.

[0262] "Preprocessing" refers to processing performed on stored personal data, and includes deleting unnecessary spaces and symbols in the case of text data, and converting audio data into text using voice recognition technology.

[0263] "Features" are useful information extracted from personal data, and include keywords, phrases, grammatical structures, speech rhythm and intonation, etc.

[0264] An "AI model" is a mathematical model built and trained using machine learning techniques to recreate the speaking style and characteristics of a deceased person.

[0265] An "emotion engine" is a technology that recognizes emotions in response to user input and generates a response that corresponds to that emotion.

[0266] An "interface" is an interactive environment that allows users to communicate with the deceased using trained AI models and emotion engines, including chatbots and voice response systems.

[0267] A "uniform format" is a state in which data has been converted into a common format through preprocessing, allowing subsequent processing to proceed smoothly.

[0268] The present invention includes a system that reproduces the characteristics and patterns of the deceased and enables communication between the deceased and the user, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. This system is realized mainly through the process of providing personal data by the user, data processing by the server, learning of an AI model, and emotion recognition and response generation.

[0269] System configuration

[0270] 1. User Interface

[0271] Users upload personal data relating to the deceased using a dedicated application or web interface, which is designed to be easy for users to navigate and guide them through the selection and upload process.

[0272] 2. Receipt and storage of data

[0273] The server receives personal data uploaded by users, encrypts the data immediately after receiving it, and stores it securely in a database or cloud storage, thus ensuring the confidentiality and integrity of the received data.

[0274] 3. Data Preprocessing

[0275] The server performs preprocessing on the stored data. For text data, unnecessary spaces and symbols are removed, and for audio data, speech recognition technology (e.g., Google Speech-to-Text API) is used to convert it into text. This preprocessing converts the data into a unified format, allowing for smooth subsequent processing.

[0276] 4. Feature Extraction

[0277] The server extracts features from the pre-processed data using natural language processing technology (e.g., spaCy). It analyzes keywords, phrases, and grammatical structures in the text to identify the deceased's style and speaking patterns. In the case of audio data, speech rhythm, intonation, and specific pronunciation patterns are also extracted.

[0278] 5. Training the Model

[0279] The server trains an AI model based on the extracted features. For example, it builds and trains a Transformer-based model using a machine learning library (e.g., TensorFlow, PyTorch). During the training process, iterative processing is performed on the provided data, and the model acquires the ability to reproduce the deceased's unique speaking style and knowledge.

[0280] 6. Emotion Recognition and Response Generation

[0281] The emotion engine recognizes the user's emotions in response to input from the user. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the AI ​​model and reflected when generating responses. This generates appropriate responses according to the user's emotional state, resulting in more human-like interactions.

[0282] 7. Providing an Interface

[0283] The server provides an interface that integrates a trained AI model and an emotion engine. Through this interface (e.g., a chatbot or voice response system), users can enjoy conversations based on the recreated features and emotion recognition of the deceased.

[0284] Specific examples

[0285] 1. User Data Provision:

[0286] Users open the application, select and upload the deceased person's diary and audio files.

[0287] 2. Receiving and storing data:

[0288] The server receives the deceased person's diary files and audio files, encrypts the data, and stores it in cloud storage.

[0289] 3. Data preprocessing:

[0290] The server cleans the diary text, removing unnecessary spaces and symbols, and converts the audio file to text using the Google Speech-to-Text API.

[0291] 4. Feature extraction:

[0292] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[0293] 5. Train the model:

[0294] The server uses TensorFlow to train a Transformer model based on the features.

[0295] 6. Emotion Recognition and Response Generation:

[0296] In response to text or voice input from the user, the server uses an emotion engine to identify the user's emotion, and the AI ​​model generates the optimal response based on that emotion.

[0297] 7. Providing an interface:

[0298] The server provides a chatbot that integrates a trained model and an emotion engine, making it accessible to users.

[0299] 8. Communication:

[0300] The user types, "How are you feeling today?" and the server's AI model recognizes the user's emotion of joy and replies, "I feel great today," in the style of the deceased.

[0301] This system can not only reproduce the characteristics and conversational style of the deceased, but also identify the user's emotions and generate appropriate responses based on those emotions, enabling richer communication.

[0302] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0303] Step 1: Provide user data

[0304] The user opens a dedicated application or web interface and selects and uploads personal data related to the deceased (e.g., diary file "diary.txt" or audio file "voice_recording.mp3").

[0305] Input: User-selected text and audio files

[0306] Output: The uploaded file is sent to the server as input.

[0307] Step 2: Receiving and storing data

[0308] The server receives the personal data uploaded by the user, encrypts it with the AES-256 algorithm, and stores it in cloud storage.

[0309] Input: User-uploaded text and audio files

[0310] Output: The encrypted data is stored in cloud storage.

[0311] Step 3: Preprocessing the data

[0312] The server loads the stored data from cloud storage. For text data, it uses regular expressions to remove unnecessary spaces and symbols. For audio data, it uses the Google Speech-to-Text API to convert the audio to text.

[0313] Input: Encrypted text and audio data stored in cloud storage

[0314] Output: Clean text data and converted text data

[0315] Step 4: Feature extraction

[0316] The server extracts features from the preprocessed data using natural language processing technology (e.g., spaCy). Specifically, it analyzes keywords, phrases, grammatical structures, and speech rhythm and intonation in the text.

[0317] Input: Preprocessed text data

[0318] Output: Extracted keywords, phrases, grammatical structures, speech rhythm, intonation, and other features

[0319] Step 5: Training the model

[0320] The server trains an AI model based on the features. It uses a machine learning library (e.g., TensorFlow, PyTorch) to build and train a Transformer-based model.

[0321] Input: extracted features

[0322] Output: Trained AI model

[0323] Step 6: Emotion recognition and response generation

[0324] The user inputs a message into the system. The emotion engine on the server analyzes the user's input message and identifies the emotional state (e.g., joy, sadness, anger, etc.). The AI ​​model generates an appropriate response based on the emotion.

[0325] Input: Input message from the user

[0326] Output: Response message generated by the AI ​​model according to the emotion

[0327] Step 7: Providing an Interface

[0328] The server provides an interface (e.g., a chatbot) that integrates a trained AI model and an emotion engine. Users can access this interface through a browser or application.

[0329] Input: trained AI model, emotion engine, user message

[0330] Output: The interface presented to the user

[0331] Step 8: Communicate

[0332] The user initiates a dialogue with the deceased, and the server analyzes the user's input and generates a response based on their emotional state. For example, if the user types, "How are you feeling today?", the server's AI model will recognize the emotion of joy and generate the response, "I feel great today."

[0333] Input: User interaction message

[0334] Output: An appropriate response message based on the sentiment

[0335] This allows the characteristics and conversation style of the deceased to be reproduced, enabling more realistic communication that responds to the user's emotions.

[0336] (Application example 2)

[0337] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] In current virtual stores, it is difficult to provide personalized product recommendations to users, and there are currently no systems that provide an emotional shopping experience based on personal data related to the deceased. Furthermore, there is a lack of interfaces that can recognize the user's emotions and respond or provide recommendations accordingly. Therefore, there is a need for a system that can recognize the user's emotional state and generate optimal responses based on that, especially for a virtual store that can provide emotionally rich communication through dialogue that reflects the characteristics of the deceased.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0340] In this invention, the server includes means for a user to upload individual data related to the deceased, means for receiving and storing the uploaded individual data, means for extracting feature data from the stored individual data, means for training a generative AI model based on the extracted feature data, means for recognizing the user's emotional state using an emotion engine, means for providing an interface that allows the user to communicate with the deceased in an emotionally adapted manner using the trained generative AI model and the emotion engine, and means for making product suggestions through dialogue in a virtual store that reflects the characteristics of the deceased. This allows the user to receive personalized product suggestions through emotion-based communication with the deceased.

[0341] "User" refers to an individual who uses the system to upload personal data related to the deceased and communicate in the virtual store.

[0342] A "deceased person" refers to an entity that had a past relationship with a user of the system, and is recreated by providing a basis for individual data related to the deceased.

[0343] "Individual data" is a general term for digital information that includes unique information related to the deceased, such as diaries, message history, and audio files.

[0344] "Uploading means" refers to the hardware and software that allows users to transfer individual data to the server.

[0345] "Means for receiving and storing" refers to the function for receiving uploaded individual data on the server side and storing it safely.

[0346] "Characteristic data" is data extracted from the stored individual data that indicates the unique characteristics of the deceased, such as their sentence structure and pronunciation patterns.

[0347] "Learning means" refers to a method and apparatus for training a generative AI model based on extracted feature data to reproduce the features of the deceased.

[0348] "Generative AI model" refers to an artificial intelligence model that is trained using machine learning algorithms to reproduce the conversational style and responses of a deceased person.

[0349] "Emotion Engine" refers to algorithms and software for recognizing a user's emotional state and generating responses based on that.

[0350] "Interface" refers to the user interaction environment that allows users to communicate with the deceased using the generative AI model and emotion engine.

[0351] A "virtual store" refers to a store-like environment that exists in a virtual space, and is a virtual platform where users can receive product suggestions and make purchases.

[0352] "Product suggestion means" refers to a method and device for reflecting the characteristics of the deceased person in a virtual store and suggesting appropriate products to the user.

[0353] This invention includes a system that reproduces the characteristics and patterns of the deceased and allows the user to communicate with the deceased, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. The system's primary purpose is to suggest products through dialogue that reflects the characteristics of the deceased in a virtual store.

[0354] Generating a Program

[0355] This system is realized based on the following program configuration.

[0356] 1. User data provision:

[0357] Using a dedicated application, users upload personal data related to the deceased, such as diary entries, message history, or audio recordings.

[0358] 2. Receiving and storing data:

[0359] The server receives the individual data uploaded by users and securely stores it in a database or cloud storage. The data is encrypted immediately after receiving it to ensure confidentiality and integrity of the data.

[0360] 3. Data preprocessing:

[0361] The server performs preprocessing on the stored data, such as deleting unnecessary spaces and symbols in the case of text data, or converting audio data into text using a speech recognition API, thereby converting the data into a unified format.

[0362] 4. Extracting feature data:

[0363] After the preprocessing is complete, the server uses natural language processing technology to extract feature data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the deceased's writing style and speaking patterns.

[0364] 5. Training the generative AI model:

[0365] After the feature data is extracted, the server trains a generative AI model based on the extracted feature data. For example, it uses a machine learning library to build and train a Transformer-based model. During the training process, iterative processing is performed on the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge.

[0366] 6. Emotion Recognition and Response Generation:

[0367] The emotion engine recognizes the user's emotions in response to user input. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the generative AI model and reflected when generating responses. This allows for the generation of appropriate responses according to the user's emotional state.

[0368] 7. Virtual store interface:

[0369] The server provides an interface that integrates a trained generative AI model and an emotion engine. Through smart glasses, users can enjoy conversations based on the deceased's recreated features and emotion recognition within a virtual store. For example, if a user types, "What do you think of this red dress?", the server's AI model will recognize the user's emotion and reply in the deceased's style, "This dress is vibrant and lovely. I think it suits you well."

[0370] Hardware and software used

[0371] Hardware: Server, smart glasses

[0372] Software: Machine learning libraries (e.g., Transformers), speech recognition APIs, databases and cloud storage, emotion engines

[0373] Specific examples

[0374] Scenario: A user wears smart glasses and enjoys shopping while walking through a virtual store.

[0375] Example dialogue:

[0376] User: "What do you think about this red dress?"

[0377] Virtual Guide: "This dress is vibrant and beautiful. I think it would look great on you."

[0378] Prompt Sentence Examples

[0379] The following text recognizes the user's emotional state when they suggest a product and generates a response based on that emotion, continuing the dialogue while reflecting the characteristics and style of the deceased.

[0380] User: What do you think about this red dress?

[0381] Emotional state: Joy

[0382] Response: This dress is vibrant and beautiful. I think it looks great on you.

[0383] In this way, the system not only reproduces the characteristics and conversational style of the deceased with high accuracy, but also identifies the user's emotions and returns appropriate responses based on those emotions, enabling richer communication.

[0384] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0385] Step 1:

[0386] The user uploads personal data relating to the deceased person using the application.

[0387] Specifically, the user selects data such as diary entries, message history, and audio files and uploads them on the application. At this time, the application generates an API request to transfer the data to the server and sends the data.

[0388] Input: Individual data such as diary entries, message history, and audio files

[0389] Output: Data transfer to the server in the form of an API request

[0390] Step 2:

[0391] The server receives the uploaded individual data, securely encrypts it, and stores it in a database or cloud storage.

[0392] Specifically, the server immediately encrypts the data it receives and stores it in a database or cloud storage, ensuring the confidentiality and integrity of the data.

[0393] Input: Individual data transferred by the user

[0394] Output: Encrypted database or cloud storage

[0395] Step 3:

[0396] The server performs preprocessing on the stored individual data.

[0397] Specifically, in the case of text data, unnecessary spaces and symbols are removed, and in the case of voice data, it is converted into text data using a voice recognition API. By converting the data into a unified format, subsequent processing becomes smoother.

[0398] Input: Data from an encrypted database or cloud storage

[0399] Output: Preprocessed data in a unified format

[0400] Step 4:

[0401] The server extracts feature data from the preprocessed data.

[0402] Specifically, natural language processing techniques are used to analyze keywords, phrases, and grammatical structures in text to identify the deceased's writing style and speaking patterns, and in the case of audio data, to extract speech rhythm, intonation, and specific pronunciation patterns.

[0403] Input: Data converted into a unified format

[0404] Output: characteristic data of the deceased

[0405] Step 5:

[0406] The server trains a generative AI model based on the extracted feature data.

[0407] Specifically, a Transformer-based AI model is built and trained using machine learning libraries (e.g., Transformers). During the training process, the model iterates on the provided data to gain the ability to reproduce the deceased person's unique speaking style and knowledge.

[0408] Input: Feature data

[0409] Output: A trained generative AI model

[0410] Step 6:

[0411] The user provides input through an interface to interact with the generative AI model.

[0412] Specifically, using smart glasses in a virtual store, users can input questions or comments by voice or text, and this data is sent to a server.

[0413] Input: User questions and comments

[0414] Output: Transfer of input data to the server

[0415] Step 7:

[0416] The server uses an emotion engine to recognize the user's emotional state and generates an appropriate response using a generative AI model.

[0417] Specifically, the emotion engine identifies the emotional state (e.g., joy, sadness, etc.) from the user's input data (voice or text) and feeds that emotion back to the generative AI model, which then generates the optimal response by taking into account the characteristics of the deceased and the user's emotions.

[0418] Input: User input data, emotion engine recognition results

[0419] Output: An appropriate response based on the user's sentiment

[0420] Step 8:

[0421] The server provides the generated response to the user in the virtual store.

[0422] Specifically, the generated response is communicated to the user through the smart glasses. For example, if a user asks, "What do you think of this red dress?", the server's AI model will generate a response such as, "This dress is vibrant and beautiful. I think it suits you well," and present it to the user through the smart glasses.

[0423] Input: The generated response

[0424] Output: Presentation of response through smart glasses

[0425] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0426] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0427] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0428] [Second embodiment]

[0429] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0430] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0431] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0432] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0433] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0434] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0435] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0436] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0437] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0438] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0439] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0440] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0441] This invention is a system that can reproduce the characteristics and patterns of a deceased person and communicate with them. This system is realized mainly by training an AI model on a server based on personal data provided by the user.

[0442] User Actions

[0443] Using a dedicated application or web interface, users upload personal data (e.g., text data, audio data) related to the deceased, such as diary entries, blogs, message histories, or audio recordings written by the deceased.

[0444] Receiving and storing data

[0445] The server receives the personal data uploaded by the user, which is then encrypted for security purposes and stored in a database or cloud storage, ensuring the confidentiality of the data.

[0446] Data Preprocessing

[0447] The server performs preprocessing on the stored data. This preprocessing includes removing unnecessary symbols and tags from text data, and converting audio data into text using a speech recognition API. This converts the data into a unified format, allowing for smooth subsequent processing.

[0448] Feature extraction

[0449] After preprocessing, the server extracts features from the data. Specifically, it uses natural language processing technology to analyze keywords, distinctive phrases, and grammatical structures in the text to identify the deceased's writing style and speaking patterns. In the case of audio data, the server also extracts speech rhythm, intonation, and specific pronunciation patterns.

[0450] Model training

[0451] After the features are extracted, the server trains an AI model based on the extracted features. For example, a Transformer-based model is built and trained using a machine learning library. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge. This trained AI model is then used to interact with the user.

[0452] Providing an interface

[0453] Once the trained AI model is complete, the server integrates it into a user-accessible interface (e.g., a chatbot or voice response system). This interface allows users to easily access and enjoy virtual conversations with the deceased. A dedicated chat application or web interface is provided, through which users can ask questions and enjoy conversations with the AI.

[0454] Specific examples

[0455] 1. User data provision:

[0456] The user opens the application, selects and uploads the diary file of the deceased person.

[0457] 2. Receiving and storing data:

[0458] The server receives the diary files of the deceased person, encrypts the data, and stores it in cloud storage.

[0459] 3. Data preprocessing:

[0460] The server cleans the diary text and removes unnecessary HTML tags.

[0461] 4. Feature extraction:

[0462] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[0463] 5. Train the model:

[0464] The server uses a machine learning library to train a Transformer model based on the features.

[0465] 6. Providing an interface:

[0466] The server integrates the trained model into the application and makes the chatbot available for users to access.

[0467] 7. Communication:

[0468] Through the application, the user types, "What's the weather like today?" and the AI ​​responds, "It's sunny today," in the style of the deceased.

[0469] In this way, the system can accurately reproduce the characteristics and conversational style of the deceased and engage in dialogue with the user, thereby enabling the knowledge and skills of the deceased to be passed on to future generations, contributing to the preservation of culture and the continuation of industry.

[0470] The processing flow will be explained below.

[0471] Step 1:

[0472] The user selects and uploads personal data related to the deceased (e.g., text files, audio files) through a dedicated application or web interface.

[0473] Step 2:

[0474] The terminal transmits the data selected by the user to the server, and before transmitting the data, it is encrypted as necessary.

[0475] Step 3:

[0476] The server receives the data sent from the device, and immediately after receiving it, it re-encrypts the data and stores it securely in a database or cloud storage.

[0477] Step 4:

[0478] The server starts preprocessing the stored data. In the case of text data, unnecessary spaces and special characters are removed and the format is standardized.

[0479] Step 5:

[0480] If voice data is included, the server converts the voice data into text data using a voice recognition API, and this converted text data also undergoes preprocessing.

[0481] Step 6:

[0482] After preprocessing is complete, the server uses natural language processing technology to extract features from the data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the characteristics of the deceased.

[0483] Step 7:

[0484] The server uses the extracted features to train an AI model using a machine learning library, for example, by building and training a Transformer-based model.

[0485] Step 8:

[0486] The trained AI model is tested and evaluated, and adjustments are made as needed. The server manages the versioning of the generated model and stores it securely.

[0487] Step 9:

[0488] The server then integrates the trained AI model into a user interface (chatbot or voice response system), allowing users to interact with the recreated features of the deceased.

[0489] Step 10:

[0490] Users enter questions or messages through the interface, which are sent to the AI ​​model on the server, which uses the model to generate an appropriate response and sends the answer back to the user.

[0491] Step 11:

[0492] The terminal displays the response received from the server to the user, and the user can continue the conversation by sending another question or message.

[0493] Example 1

[0494] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0495] It is necessary to provide a system that can reproduce the characteristics and communication style of the deceased and converse with them while ensuring the security of personal data. It is also difficult to deal with various forms of personal data, including voice data, and to effectively train an AI model to realize the dialogue.

[0496] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0497] In this invention, the server includes means for a user to upload personal data related to the deceased, means for receiving and storing the uploaded personal data, means for encrypting the personal data and storing it in cloud storage or a database, means for preprocessing the stored personal data and converting it into a unified format, means for extracting features from the preprocessed personal data, means for training an AI model based on the extracted features, means for providing an interface using the trained AI model to allow the user to communicate with the deceased, and means for responding to the user's questions and messages through the provided interface. This makes it possible to extract features from various forms of data about the deceased and effectively train the AI ​​model to realize dialogue while ensuring the security of the personal data.

[0498] "Uploading" is the act of a user sending data from their device to a server.

[0499] "Personal data" refers to all information generated by an individual, including text data and audio data relating to a deceased person.

[0500] "Encryption" is the process of converting data using a specific algorithm to make it unreadable to third parties.

[0501] "Cloud storage" is an external storage service that stores and accesses data over the Internet.

[0502] "Preprocessing" is the process of cleaning and formatting data to make it easier for subsequent processing.

[0503] "Features" are important elements or parameters extracted from data, and are information used to train AI models.

[0504] An "AI model" is a learning algorithmic structure built using artificial intelligence techniques and trained to perform a specific task.

[0505] "Interface" refers to the tools and environments through which users interact with a system, including chatbots and voice response systems.

[0506] "Voice recognition technology" is a technology that analyzes voice data and converts it into text.

[0507] A "Transformer model" is a type of deep learning model used in natural language processing, capable of analyzing the meaning and context of text.

[0508] The present invention is a system that can reproduce the characteristics and patterns of a deceased person and communicate with the deceased. To implement this system, the entire system is constructed based on the following procedure.

[0509] First, users use a dedicated application or web interface to upload personal data related to the deceased, including diary entries, blogs, message history, or audio recordings.

[0510] The server then receives the personal data uploaded by the user, encrypts the data, and stores it in cloud storage or a database. The received data is encrypted using the Advanced Encryption Standard (AES) to ensure security. The encryption key is stored in a separate, secure location.

[0511] The server preprocesses the stored data. For text data, unnecessary symbols and HTML tags are removed and the data is normalized. For audio data, the data is converted to text using Google Cloud Speech-to-Text or other speech recognition APIs.

[0512] After preprocessing, the server extracts features from the data. For example, natural language processing techniques such as TF-IDF (Term Frequency-Inverse Document Frequency) and Word2Vec are used to extract important keywords and unique phrases from the text. In the case of audio data, features such as speech rhythm, intonation, and specific pronunciation patterns are also extracted.

[0513] After the features are extracted, the server trains an AI model based on them. It uses machine learning libraries such as TensorFlow and PyTorch to build a Transformer-based model, using the extracted features as training data. Through an iterative process, it learns the deceased person's unique speaking style and knowledge.

[0514] Using the trained AI model, the server provides a user-accessible interface, consisting of a chatbot and a voice response system, allowing users to enjoy virtual conversations with the deceased through a dedicated chat application or web interface.

[0515] As a specific example of operation, a user opens the application, selects and uploads the diary file of the deceased person. The server then encrypts, stores, and preprocesses the file, extracts features, and trains the AI ​​model. Finally, when the user asks through the application, "What's the weather like today?", the AI ​​responds, "It's sunny today," in the style of the deceased person.

[0516] This system can accurately reproduce the characteristics and conversational style of the deceased and enable dialogue with the user, thereby passing on the knowledge and skills of the deceased to the next generation and contributing to the preservation of culture and the continuation of industry.

[0517] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0518] Step 1: Upload user data

[0519] The user opens a dedicated application or web interface, selects a data file (e.g., diary, audio file) related to the deceased, and clicks the upload button. The input data is the personal data selected by the user, and the output is to send the selected file to the server. Specifically, the user clicks the file selection button in the application, selects "diary.txt," and presses the upload button.

[0520] Step 2: Receiving and Encrypting Data

[0521] The server stores the received personal data in a temporary folder. It then encrypts the received data using the AES (Advanced Encryption Standard) algorithm. The input data is the raw personal data sent by the user, and the output is the encrypted data. Specifically, the server receives "diary.txt," encrypts it using the AES algorithm, and saves it in an AWS S3 bucket as "encrypted_diary.txt."

[0522] Step 3: Preprocessing the data

[0523] The server loads the stored data and begins preprocessing. The input data is encrypted personal data, and the output is cleaned and normalized data. The processing involves removing unnecessary symbols and HTML tags from text data, and converting audio data to text using Google Cloud Speech-to-Text. Specifically, it loads "encrypted_diary.txt," decrypts it, and then removes HTML tags and special characters.

[0524] Step 4: Feature extraction

[0525] The server extracts features from the data after preprocessing. The input data is cleaned personal data, and the output is the extracted features. Specific techniques include using natural language processing techniques (e.g., TF-IDF, Word2Vec) to extract important keywords and distinctive phrases from the text. In the case of audio data, features of audio patterns are extracted. Specific operations include extracting feature keywords from "diary_clean.txt" using TF-IDF and saving them in "keyword_list.txt."

[0526] Step 5: Training the AI ​​model

[0527] The server builds an AI model based on the features and trains it. The input data are the extracted features, and the output is a trained AI model. Specifically, a Transformer-based model is created using TensorFlow or PyTorch, and specific features are used as training data. Specifically, a Transformer model is built using TensorFlow, and the extracted "keyword_list.txt" is used as training data to train the model.

[0528] Step 6: Providing an Interface

[0529] The server uses the trained AI model to provide an interface that users can access. The input data is input from the user (text or voice), and the output is the response from the AI ​​model. Specifically, the trained model is integrated into a Flask web application to provide a chat interface. Users can interact with the AI ​​through a dedicated chat application or web interface.

[0530] Step 7: Communicate with users

[0531] The user inputs questions or messages through the interface, and the server responds using a trained AI model. The input data is the user's question or message, and the output is the response from the AI ​​model. Specifically, when a user types "What's the weather like today?" into a chat application, the server responds "It's sunny today" in the style of the deceased.

[0532] (Application example 1)

[0533] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0534] Currently, there are limited ways to reminisce about memories with deceased family and friends. Traditional photo albums and videos are merely static records and do not allow for interactive communication with the deceased. While there are technologies that utilize personal data to recreate the characteristics and speaking style of the deceased, there is a lack of effective integration of advanced natural language processing and voice reproduction technologies.

[0535] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0536] In this invention, the server includes: means for a user to upload personal data related to the deceased; means for receiving and storing the uploaded personal data; means for extracting features from the stored personal data; means for training a generative AI model based on the extracted features; means for providing an interface that allows the user to communicate with the deceased using the trained AI model; means for reproducing the voice and conversation style of the deceased based on the responses of the generative AI model; means for inputting prompts that the user can use to enjoy a conversation with the deceased; and means for performing preprocessing to convert voice data into text data. This enables two-way virtual communication with the deceased, allowing the user to reminisce about their memories in a more realistic and moving way.

[0537] "Means by which users can upload personal data relating to a deceased person" refers to devices or software that allow text and audio data relating to a deceased person to be sent digitally to a server.

[0538] "Means for receiving and storing uploaded personal data" refers to the technology or system that receives submitted personal data and stores it in a secure location.

[0539] "Means for extracting features from stored personal data" refers to algorithms or processes that analyze stored data and identify the characteristics and speech patterns of the deceased.

[0540] "Means for training a generative AI model" refers to the process of feeding data to an AI model using the extracted features to train the model to have the ability to reproduce the speech pattern and characteristics of the deceased.

[0541] "Means for providing an interface" means a user interface that allows a user to interact with an AI model, such as a chat application or a voice response system.

[0542] "Means of recreating the voice and speaking style of the deceased based on the response of a generative AI model" refers to technologies and systems that synthesize the voice and distinctive speaking style of the deceased based on the output of an AI model.

[0543] "Means for inputting prompt text" refers to a device or interface that allows a user to input text to ask or speak to the deceased.

[0544] The "means for performing preprocessing to convert voice data into text data" refers to a technology for generating text from a voice file, such as a voice recognition system.

[0545] This invention is a system that allows users to upload personal data related to the deceased, enabling communication that recreates the characteristics and patterns of the deceased.

[0546] 1. Program Generation

[0547] The user uses their device to upload text and audio data related to the deceased to the server. The server receives this data and stores it in a secure location. The server also extracts features from the stored data and analyzes the deceased's unique speaking style and expressions. Based on these features, the server trains a generative AI model to build a model that mimics the characteristics of the deceased. The server then uses the trained AI model to provide an interface that allows the user to interact with the deceased.

[0548] 2. Explanation of program processing

[0549] This system uses the following major hardware and software:

[0550] Server: Used for data storage and computation. A good example is Amazon EC2.

[0551] Generative AI model: Uses OpenAI API to recreate the speaking style of the deceased.

[0552] Web framework: Django is used to provide the user interface.

[0553] Data processing and calculation:

[0554] The server receives and stores the uploaded personal data. It then preprocesses the stored data and converts it into a unified format. For example, it uses a speech-to-text API to convert voice data into text data. It then uses natural language processing technology to extract features and trains a generative AI model. Once trained, the AI ​​model can reproduce the deceased person's unique speaking style.

[0555] 3. Specific Examples

[0556] Below are some specific scenarios:

[0557] 1. The user opens the application and uploads an audio file of the deceased person.

[0558] 2. The server receives the audio file and stores it in secure cloud storage.

[0559] 3. Preprocess the audio file and convert it to text, for example, using the Google Speech-to-Text API.

[0560] 4. The server analyzes the text data and extracts keywords and phrases characteristic of the deceased.

[0561] 5. Based on the extracted features, a generative AI model is trained using the OpenAI API.

[0562] 6. Using the learning model, users can enjoy interacting with their deceased loved ones through chat applications.

[0563] 7. When the user types "Hello, Grandpa," the system responds by recreating the deceased's distinctive speaking style, "Hello, how are you?"

[0564] Prompt Sentence Examples

[0565] "Grandpa, how was your day today?"

[0566] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0567] Step 1: The user uses a device to upload text and audio data related to the deceased person to the server.

[0568] Input: Text data and audio data

[0569] Specific operation: The user uses the application's upload function to select and send a note or audio file of the deceased person, which is then received by the server.

[0570] Step 2: The server receives the uploaded personal data and stores it in a secure location.

[0571] Input: Text and audio data uploaded by users

[0572] Output: Stored personal data

[0573] What it does: Encrypts received data and stores it in cloud storage. Uses security protocols to ensure data confidentiality.

[0574] Step 3: The server preprocesses the stored data and converts it into a unified format.

[0575] Input: Stored personal data

[0576] Output: Formatted text data

[0577] Specific operations: For text data, remove unnecessary tags and symbols. For audio data, convert audio to text using the Speech-to-Text API.

[0578] Step 4: The server extracts features from the preprocessed data.

[0579] Input: Formatted text data

[0580] Output: Extracted features

[0581] What it does: It uses natural language processing technology to analyze keywords and distinctive phrases in texts to identify patterns in the speech and writing style of the deceased.

[0582] Step 5: The server trains a generative AI model based on the extracted features.

[0583] Input: Extracted features

[0584] Output: A trained generative AI model

[0585] What it does: It uses a machine learning library (e.g., Transformer) to train an AI model, which iterates to learn the deceased person's speaking style.

[0586] Step 6: The server uses the trained AI model to provide an interface that allows the user to interact with the deceased.

[0587] Input: A trained generative AI model

[0588] Output: User interface

[0589] What it does: Build an interface that acts as a chatbot or voice response system and provides it to users in an easily accessible way.

[0590] Step 7: The user uses the terminal to enter a prompt and interact with the deceased.

[0591] Input: prompt statement

[0592] Output: Response text or audio

[0593] How it works: When a user enters a prompt phrase, such as "Hello, Grandpa," the server uses a generative AI model to generate a response in the style of the deceased person, and returns it as text or voice.

[0594] Through these steps, the system can accurately reproduce the characteristics and conversation style of the deceased person and engage in a virtual dialogue with the user.

[0595] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0596] The present invention includes a system that reproduces the characteristics and patterns of the deceased and enables communication between the deceased and the user, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. This system is realized mainly through the process of providing personal data by the user, data processing by the server, learning of an AI model, and emotion recognition and response generation.

[0597] User operations

[0598] Using a dedicated application or web interface, users select and upload personal data (e.g., text files, audio files) related to the deceased, such as diary entries, blogs, message histories, or audio recordings written by the deceased.

[0599] Receiving and storing data

[0600] The server receives the personal data uploaded by the user, encrypts the data immediately after receiving it, and stores it securely in a database or cloud storage, ensuring the confidentiality and integrity of the data.

[0601] Data Preprocessing

[0602] The server performs preprocessing on the stored data. This preprocessing includes removing unnecessary spaces and symbols in the case of text data, and converting audio data into text using a speech recognition API. This converts the data into a unified format, allowing for smooth subsequent processing.

[0603] Feature extraction

[0604] After preprocessing is complete, the server uses natural language processing technology to extract features from the data. Specifically, it analyzes keywords, phrases, and grammatical structures in the text to identify the deceased's writing style and speaking patterns. In the case of audio data, it also extracts the rhythm and intonation of speech, as well as specific pronunciation patterns.

[0605] Model training

[0606] After the features are extracted, the server trains an AI model based on the extracted features. For example, a Transformer-based model is built and trained using a machine learning library. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge. This trained AI model is then used to interact with the user.

[0607] Emotion Recognition and Response Generation

[0608] The emotion engine recognizes the user's emotions in response to input from the user. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the AI ​​model and reflected when generating responses. This allows for the generation of appropriate responses according to the user's emotional state, resulting in more human-like interactions.

[0609] Providing an interface

[0610] The server provides users with an interface (e.g., a chatbot or voice response system) that integrates the trained AI model and emotion engine. Through this interface, users can enjoy conversations based on the recreated features and emotion recognition of the deceased person.

[0611] Specific examples

[0612] 1. User Data Provision:

[0613] The user opens the application, selects and uploads the diary and audio files of the deceased person.

[0614] 2. Receiving and storing data:

[0615] The server receives the deceased person's diary files and audio files, encrypts the data, and stores it in cloud storage.

[0616] 3. Data preprocessing:

[0617] The server cleans the diary text, removing unnecessary spaces and symbols, and converts the audio file into text using a speech recognition API.

[0618] 4. Feature extraction:

[0619] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[0620] 5. Train the model:

[0621] The server uses a machine learning library to train a Transformer model based on the features.

[0622] 6. Emotion Recognition and Response Generation:

[0623] In response to text or voice input from the user, the server uses an emotion engine to identify the user's emotion, and the AI ​​model generates the optimal response based on that emotion.

[0624] 7. Providing an interface:

[0625] The server provides a chatbot that integrates a trained model and an emotion engine, making it accessible to users.

[0626] 8. Communication:

[0627] The user types, "How are you feeling today?" and the server's AI model recognizes the user's emotion of joy and replies, "I feel great today," in the style of the deceased.

[0628] In this way, the system can not only accurately reproduce the characteristics and conversational style of the deceased, but also identify the user's emotions and provide appropriate responses based on those emotions, enabling richer communication.

[0629] The processing flow will be explained below.

[0630] Step 1:

[0631] The user selects and uploads personal data related to the deceased (e.g., text files, audio files) through a dedicated application or web interface.

[0632] Step 2:

[0633] The terminal transmits the data selected by the user to the server, and before transmitting the data, it is encrypted as necessary.

[0634] Step 3:

[0635] The server receives the data sent from the device, then immediately encrypts it again and stores it securely in a database or cloud storage.

[0636] Step 4:

[0637] The server starts preprocessing the stored data. For text data, unnecessary spaces and special characters are deleted and the format is standardized. For audio data, a speech recognition API is used to convert it into text data.

[0638] Step 5:

[0639] The server uses natural language processing technology to extract features from the preprocessed data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the characteristics of the deceased. In the case of audio data, it also extracts the rhythm and intonation of speech, as well as specific pronunciation patterns.

[0640] Step 6:

[0641] The server uses the extracted features to train an AI model using a machine learning library, such as building and training a Transformer-based model. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge.

[0642] Step 7:

[0643] The server uses an emotion engine to identify the emotion of user input (text and voice). The emotion engine performs text and voice analysis to determine emotional states such as joy, sadness, and anger.

[0644] Step 8:

[0645] The server adjusts the AI ​​model's response based on the emotional information obtained from the emotion engine. By incorporating the emotional information into response generation, an appropriate response is generated according to the user's emotional state.

[0646] Step 9:

[0647] The server provides an interface that integrates trained AI models and emotion engines. For example, users can interact with AI through chatbots or voice response systems.

[0648] Step 10:

[0649] When a user enters a question or message through the interface, it is sent to the server's AI model and emotion engine. The server uses the emotion engine to identify the user's emotion and uses the AI ​​model to generate an appropriate response. This response is then sent back to the user.

[0650] Step 11:

[0651] The device displays the response received from the server to the user. The user can also send questions or messages again, allowing for a continuous dialogue. This allows the user to experience a more human-like dialogue through an AI model that recognizes emotions and recreates the characteristics of the deceased.

[0652] Example 2

[0653] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0654] A system that enables conversation with the deceased must accurately reproduce the characteristics and conversational style of the deceased, recognize the user's emotions, and reflect them in the conversation. It must also convert various data formats into a unified format and efficiently extract features while ensuring the confidentiality and integrity of personal data.

[0655] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0656] In this invention, the server includes: a means for a user to upload personal data related to the deceased; a means for receiving and saving the uploaded personal data; a means for preprocessing the saved personal data, such as deleting unnecessary spaces and symbols in the case of text data and converting audio data into text using speech recognition technology; a means for extracting features from the preprocessed data; a means for training an AI model based on the extracted features to reproduce the deceased's unique conversational style; a means for recognizing emotions in response to user input and generating appropriate responses according to those emotions; and a means for providing an interface that allows the user to communicate with the deceased using the trained AI model and emotion engine. This makes it possible to reproduce the deceased's characteristics and conversational style while generating responses according to the user's emotions.

[0657] A "user" is a person who wishes to use the system to interact with the deceased.

[0658] A "deceased person" is someone who has already passed away, and the system will recreate that person's characteristics and conversation style.

[0659] "Personal data" is information related to a deceased person that is uploaded by a user, including text files, audio files, diaries, message history, etc.

[0660] "Means for uploading" refers to the method by which a user provides personal data relating to the deceased person to the system, including a dedicated application or web interface.

[0661] "Means of receiving and storing" refers to the process by which the server receives the uploaded personal data and stores it safely.

[0662] "Preprocessing" refers to processing performed on stored personal data, and includes deleting unnecessary spaces and symbols in the case of text data, and converting audio data into text using voice recognition technology.

[0663] "Features" are useful information extracted from personal data, and include keywords, phrases, grammatical structures, speech rhythm and intonation, etc.

[0664] An "AI model" is a mathematical model built and trained using machine learning techniques to recreate the speaking style and characteristics of a deceased person.

[0665] An "emotion engine" is a technology that recognizes emotions in response to user input and generates a response that corresponds to that emotion.

[0666] An "interface" is an interactive environment that allows users to communicate with the deceased using trained AI models and emotion engines, including chatbots and voice response systems.

[0667] A "uniform format" is a state in which data has been converted into a common format through preprocessing, allowing subsequent processing to proceed smoothly.

[0668] The present invention includes a system that reproduces the characteristics and patterns of the deceased and enables communication between the deceased and the user, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. This system is realized mainly through the process of providing personal data by the user, data processing by the server, learning of an AI model, and emotion recognition and response generation.

[0669] System configuration

[0670] 1. User Interface

[0671] Users upload personal data relating to the deceased using a dedicated application or web interface, which is designed to be easy for users to navigate and guide them through the selection and upload process.

[0672] 2. Receipt and storage of data

[0673] The server receives personal data uploaded by users, encrypts the data immediately after receiving it, and stores it securely in a database or cloud storage, thus ensuring the confidentiality and integrity of the received data.

[0674] 3. Data Preprocessing

[0675] The server performs preprocessing on the stored data. For text data, unnecessary spaces and symbols are removed, and for audio data, speech recognition technology (e.g., Google Speech-to-Text API) is used to convert it into text. This preprocessing converts the data into a unified format, allowing for smooth subsequent processing.

[0676] 4. Feature Extraction

[0677] The server extracts features from the pre-processed data using natural language processing technology (e.g., spaCy). It analyzes keywords, phrases, and grammatical structures in the text to identify the deceased's style and speaking patterns. In the case of audio data, speech rhythm, intonation, and specific pronunciation patterns are also extracted.

[0678] 5. Training the Model

[0679] The server trains an AI model based on the extracted features. For example, it builds and trains a Transformer-based model using a machine learning library (e.g., TensorFlow, PyTorch). During the training process, iterative processing is performed on the provided data, and the model acquires the ability to reproduce the deceased's unique speaking style and knowledge.

[0680] 6. Emotion Recognition and Response Generation

[0681] The emotion engine recognizes the user's emotions in response to input from the user. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the AI ​​model and reflected when generating responses. This generates appropriate responses according to the user's emotional state, resulting in more human-like interactions.

[0682] 7. Providing an Interface

[0683] The server provides an interface that integrates a trained AI model and an emotion engine. Through this interface (e.g., a chatbot or voice response system), users can enjoy conversations based on the recreated features and emotion recognition of the deceased.

[0684] Specific examples

[0685] 1. User Data Provision:

[0686] Users open the application, select and upload the deceased person's diary and audio files.

[0687] 2. Receiving and storing data:

[0688] The server receives the deceased person's diary files and audio files, encrypts the data, and stores it in cloud storage.

[0689] 3. Data preprocessing:

[0690] The server cleans the diary text, removing unnecessary spaces and symbols, and converts the audio file to text using the Google Speech-to-Text API.

[0691] 4. Feature extraction:

[0692] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[0693] 5. Train the model:

[0694] The server uses TensorFlow to train a Transformer model based on the features.

[0695] 6. Emotion Recognition and Response Generation:

[0696] In response to text or voice input from the user, the server uses an emotion engine to identify the user's emotion, and the AI ​​model generates the optimal response based on that emotion.

[0697] 7. Providing an interface:

[0698] The server provides a chatbot that integrates a trained model and an emotion engine, making it accessible to users.

[0699] 8. Communication:

[0700] The user types, "How are you feeling today?" and the server's AI model recognizes the user's emotion of joy and replies, "I feel great today," in the style of the deceased.

[0701] This system can not only reproduce the characteristics and conversational style of the deceased, but also identify the user's emotions and generate appropriate responses based on those emotions, enabling richer communication.

[0702] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0703] Step 1: Provide user data

[0704] The user opens a dedicated application or web interface and selects and uploads personal data related to the deceased (e.g., diary file "diary.txt" or audio file "voice_recording.mp3").

[0705] Input: User-selected text and audio files

[0706] Output: The uploaded file is sent to the server as input.

[0707] Step 2: Receiving and storing data

[0708] The server receives the personal data uploaded by the user, encrypts it with the AES-256 algorithm, and stores it in cloud storage.

[0709] Input: User-uploaded text and audio files

[0710] Output: The encrypted data is stored in cloud storage.

[0711] Step 3: Preprocessing the data

[0712] The server loads the stored data from cloud storage. For text data, it uses regular expressions to remove unnecessary spaces and symbols. For audio data, it uses the Google Speech-to-Text API to convert the audio to text.

[0713] Input: Encrypted text and audio data stored in cloud storage

[0714] Output: Clean text data and converted text data

[0715] Step 4: Feature extraction

[0716] The server extracts features from the preprocessed data using natural language processing technology (e.g., spaCy). Specifically, it analyzes keywords, phrases, grammatical structures, and speech rhythm and intonation in the text.

[0717] Input: Preprocessed text data

[0718] Output: Extracted keywords, phrases, grammatical structures, speech rhythm, intonation, and other features

[0719] Step 5: Training the model

[0720] The server trains an AI model based on the features. It uses a machine learning library (e.g., TensorFlow, PyTorch) to build and train a Transformer-based model.

[0721] Input: extracted features

[0722] Output: Trained AI model

[0723] Step 6: Emotion recognition and response generation

[0724] The user inputs a message into the system. The emotion engine on the server analyzes the user's input message and identifies the emotional state (e.g., joy, sadness, anger, etc.). The AI ​​model generates an appropriate response based on the emotion.

[0725] Input: Input message from the user

[0726] Output: Response message generated by the AI ​​model according to the emotion

[0727] Step 7: Providing an Interface

[0728] The server provides an interface (e.g., a chatbot) that integrates a trained AI model and an emotion engine. Users can access this interface through a browser or application.

[0729] Input: trained AI model, emotion engine, user message

[0730] Output: The interface presented to the user

[0731] Step 8: Communicate

[0732] The user initiates a dialogue with the deceased, and the server analyzes the user's input and generates a response based on their emotional state. For example, if the user types, "How are you feeling today?", the server's AI model will recognize the emotion of joy and generate the response, "I feel great today."

[0733] Input: User interaction message

[0734] Output: An appropriate response message based on the sentiment

[0735] This allows the characteristics and conversation style of the deceased to be reproduced, enabling more realistic communication that responds to the user's emotions.

[0736] (Application example 2)

[0737] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0738] In current virtual stores, it is difficult to provide personalized product recommendations to users, and there are currently no systems that provide an emotional shopping experience based on personal data related to the deceased. Furthermore, there is a lack of interfaces that can recognize the user's emotions and respond or provide recommendations accordingly. Therefore, there is a need for a system that can recognize the user's emotional state and generate optimal responses based on that, especially for a virtual store that can provide emotionally rich communication through dialogue that reflects the characteristics of the deceased.

[0739] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0740] In this invention, the server includes means for a user to upload individual data related to the deceased, means for receiving and storing the uploaded individual data, means for extracting feature data from the stored individual data, means for training a generative AI model based on the extracted feature data, means for recognizing the user's emotional state using an emotion engine, means for providing an interface that allows the user to communicate with the deceased in an emotionally adapted manner using the trained generative AI model and the emotion engine, and means for making product suggestions through dialogue in a virtual store that reflects the characteristics of the deceased. This allows the user to receive personalized product suggestions through emotion-based communication with the deceased.

[0741] "User" refers to an individual who uses the system to upload personal data related to the deceased and communicate in the virtual store.

[0742] A "deceased person" refers to an entity that had a past relationship with a user of the system, and is recreated by providing a basis for individual data related to the deceased.

[0743] "Individual data" is a general term for digital information that includes unique information related to the deceased, such as diaries, message history, and audio files.

[0744] "Uploading means" refers to the hardware and software that allows users to transfer individual data to the server.

[0745] "Means for receiving and storing" refers to the function for receiving uploaded individual data on the server side and storing it safely.

[0746] "Characteristic data" is data extracted from the stored individual data that indicates the unique characteristics of the deceased, such as their sentence structure and pronunciation patterns.

[0747] "Learning means" refers to a method and apparatus for training a generative AI model based on extracted feature data to reproduce the features of the deceased.

[0748] "Generative AI model" refers to an artificial intelligence model that is trained using machine learning algorithms to reproduce the conversational style and responses of a deceased person.

[0749] "Emotion Engine" refers to algorithms and software for recognizing a user's emotional state and generating responses based on that.

[0750] "Interface" refers to the user interaction environment that allows users to communicate with the deceased using the generative AI model and emotion engine.

[0751] A "virtual store" refers to a store-like environment that exists in a virtual space, and is a virtual platform where users can receive product suggestions and make purchases.

[0752] "Product suggestion means" refers to a method and device for reflecting the characteristics of the deceased person in a virtual store and suggesting appropriate products to the user.

[0753] This invention includes a system that reproduces the characteristics and patterns of the deceased and allows the user to communicate with the deceased, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. The system's primary purpose is to suggest products through dialogue that reflects the characteristics of the deceased in a virtual store.

[0754] Generating a Program

[0755] This system is realized based on the following program configuration.

[0756] 1. User data provision:

[0757] Using a dedicated application, users upload personal data related to the deceased, such as diary entries, message history, or audio recordings.

[0758] 2. Receiving and storing data:

[0759] The server receives the individual data uploaded by users and securely stores it in a database or cloud storage. The data is encrypted immediately after receiving it to ensure confidentiality and integrity of the data.

[0760] 3. Data preprocessing:

[0761] The server performs preprocessing on the stored data, such as deleting unnecessary spaces and symbols in the case of text data, or converting audio data into text using a speech recognition API, thereby converting the data into a unified format.

[0762] 4. Extracting feature data:

[0763] After the preprocessing is complete, the server uses natural language processing technology to extract feature data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the deceased's writing style and speaking patterns.

[0764] 5. Training the generative AI model:

[0765] After the feature data is extracted, the server trains a generative AI model based on the extracted feature data. For example, it uses a machine learning library to build and train a Transformer-based model. During the training process, iterative processing is performed on the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge.

[0766] 6. Emotion Recognition and Response Generation:

[0767] The emotion engine recognizes the user's emotions in response to user input. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the generative AI model and reflected when generating responses. This allows for the generation of appropriate responses according to the user's emotional state.

[0768] 7. Virtual store interface:

[0769] The server provides an interface that integrates a trained generative AI model and an emotion engine. Through smart glasses, users can enjoy conversations based on the deceased's recreated features and emotion recognition within a virtual store. For example, if a user types, "What do you think of this red dress?", the server's AI model will recognize the user's emotion and reply in the deceased's style, "This dress is vibrant and lovely. I think it suits you well."

[0770] Hardware and software used

[0771] Hardware: Server, smart glasses

[0772] Software: Machine learning libraries (e.g., Transformers), speech recognition APIs, databases and cloud storage, emotion engines

[0773] Specific examples

[0774] Scenario: A user wears smart glasses and enjoys shopping while walking through a virtual store.

[0775] Example dialogue:

[0776] User: "What do you think about this red dress?"

[0777] Virtual Guide: "This dress is vibrant and beautiful. I think it would look great on you."

[0778] Prompt Sentence Examples

[0779] The following text recognizes the user's emotional state when they suggest a product and generates a response based on that emotion, continuing the dialogue while reflecting the characteristics and style of the deceased.

[0780] User: What do you think about this red dress?

[0781] Emotional state: Joy

[0782] Response: This dress is vibrant and beautiful. I think it looks great on you.

[0783] In this way, the system not only reproduces the characteristics and conversational style of the deceased with high accuracy, but also identifies the user's emotions and returns appropriate responses based on those emotions, enabling richer communication.

[0784] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0785] Step 1:

[0786] The user uploads personal data relating to the deceased person using the application.

[0787] Specifically, the user selects data such as diary entries, message history, and audio files and uploads them on the application. At this time, the application generates an API request to transfer the data to the server and sends the data.

[0788] Input: Individual data such as diary entries, message history, and audio files

[0789] Output: Data transfer to the server in the form of an API request

[0790] Step 2:

[0791] The server receives the uploaded individual data, securely encrypts it, and stores it in a database or cloud storage.

[0792] Specifically, the server immediately encrypts the data it receives and stores it in a database or cloud storage, ensuring the confidentiality and integrity of the data.

[0793] Input: Individual data transferred by the user

[0794] Output: Encrypted database or cloud storage

[0795] Step 3:

[0796] The server performs preprocessing on the stored individual data.

[0797] Specifically, in the case of text data, unnecessary spaces and symbols are removed, and in the case of voice data, it is converted into text data using a voice recognition API. By converting the data into a unified format, subsequent processing becomes smoother.

[0798] Input: Data from an encrypted database or cloud storage

[0799] Output: Preprocessed data in a unified format

[0800] Step 4:

[0801] The server extracts feature data from the preprocessed data.

[0802] Specifically, natural language processing techniques are used to analyze keywords, phrases, and grammatical structures in text to identify the deceased's writing style and speaking patterns, and in the case of audio data, to extract speech rhythm, intonation, and specific pronunciation patterns.

[0803] Input: Data converted into a unified format

[0804] Output: characteristic data of the deceased

[0805] Step 5:

[0806] The server trains a generative AI model based on the extracted feature data.

[0807] Specifically, a Transformer-based AI model is built and trained using machine learning libraries (e.g., Transformers). During the training process, the model iterates on the provided data to gain the ability to reproduce the deceased person's unique speaking style and knowledge.

[0808] Input: Feature data

[0809] Output: A trained generative AI model

[0810] Step 6:

[0811] The user provides input through an interface to interact with the generative AI model.

[0812] Specifically, using smart glasses in a virtual store, users can input questions or comments by voice or text, and this data is sent to a server.

[0813] Input: User questions and comments

[0814] Output: Transfer of input data to the server

[0815] Step 7:

[0816] The server uses an emotion engine to recognize the user's emotional state and generates an appropriate response using a generative AI model.

[0817] Specifically, the emotion engine identifies the emotional state (e.g., joy, sadness, etc.) from the user's input data (voice or text) and feeds that emotion back to the generative AI model, which then generates the optimal response by taking into account the characteristics of the deceased and the user's emotions.

[0818] Input: User input data, emotion engine recognition results

[0819] Output: An appropriate response based on the user's sentiment

[0820] Step 8:

[0821] The server provides the generated response to the user in the virtual store.

[0822] Specifically, the generated response is communicated to the user through the smart glasses. For example, if a user asks, "What do you think of this red dress?", the server's AI model will generate a response such as, "This dress is vibrant and beautiful. I think it suits you well," and present it to the user through the smart glasses.

[0823] Input: The generated response

[0824] Output: Presentation of response through smart glasses

[0825] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0826] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0827] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0828] [Third embodiment]

[0829] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0830] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0831] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0832] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0833] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0834] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0835] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0836] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0837] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0838] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0839] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0840] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0841] This invention is a system that can reproduce the characteristics and patterns of a deceased person and communicate with them. This system is realized mainly by training an AI model on a server based on personal data provided by the user.

[0842] User Actions

[0843] Using a dedicated application or web interface, users upload personal data (e.g., text data, audio data) related to the deceased, such as diary entries, blogs, message histories, or audio recordings written by the deceased.

[0844] Receiving and storing data

[0845] The server receives the personal data uploaded by the user, which is then encrypted for security purposes and stored in a database or cloud storage, ensuring the confidentiality of the data.

[0846] Data Preprocessing

[0847] The server performs preprocessing on the stored data. This preprocessing includes removing unnecessary symbols and tags from text data, and converting audio data into text using a speech recognition API. This converts the data into a unified format, allowing for smooth subsequent processing.

[0848] Feature extraction

[0849] After preprocessing, the server extracts features from the data. Specifically, it uses natural language processing technology to analyze keywords, distinctive phrases, and grammatical structures in the text to identify the deceased's writing style and speaking patterns. In the case of audio data, the server also extracts speech rhythm, intonation, and specific pronunciation patterns.

[0850] Model training

[0851] After the features are extracted, the server trains an AI model based on the extracted features. For example, a Transformer-based model is built and trained using a machine learning library. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge. This trained AI model is then used to interact with the user.

[0852] Providing an interface

[0853] Once the trained AI model is complete, the server integrates it into a user-accessible interface (e.g., a chatbot or voice response system). This interface allows users to easily access and enjoy virtual conversations with the deceased. A dedicated chat application or web interface is provided, through which users can ask questions and enjoy conversations with the AI.

[0854] Specific examples

[0855] 1. User data provision:

[0856] The user opens the application, selects and uploads the diary file of the deceased person.

[0857] 2. Receiving and storing data:

[0858] The server receives the diary files of the deceased person, encrypts the data, and stores it in cloud storage.

[0859] 3. Data preprocessing:

[0860] The server cleans the diary text and removes unnecessary HTML tags.

[0861] 4. Feature extraction:

[0862] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[0863] 5. Train the model:

[0864] The server uses a machine learning library to train a Transformer model based on the features.

[0865] 6. Providing an interface:

[0866] The server integrates the trained model into the application and makes the chatbot available for users to access.

[0867] 7. Communication:

[0868] Through the application, the user types, "What's the weather like today?" and the AI ​​responds, "It's sunny today," in the style of the deceased.

[0869] In this way, the system can accurately reproduce the characteristics and conversational style of the deceased and engage in dialogue with the user, thereby enabling the knowledge and skills of the deceased to be passed on to future generations, contributing to the preservation of culture and the continuation of industry.

[0870] The processing flow will be explained below.

[0871] Step 1:

[0872] The user selects and uploads personal data related to the deceased (e.g., text files, audio files) through a dedicated application or web interface.

[0873] Step 2:

[0874] The terminal transmits the data selected by the user to the server, and before transmitting the data, it is encrypted as necessary.

[0875] Step 3:

[0876] The server receives the data sent from the device, and immediately after receiving it, it re-encrypts the data and stores it securely in a database or cloud storage.

[0877] Step 4:

[0878] The server starts preprocessing the stored data. In the case of text data, unnecessary spaces and special characters are removed and the format is standardized.

[0879] Step 5:

[0880] If voice data is included, the server converts the voice data into text data using a voice recognition API, and this converted text data also undergoes preprocessing.

[0881] Step 6:

[0882] After preprocessing is complete, the server uses natural language processing technology to extract features from the data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the characteristics of the deceased.

[0883] Step 7:

[0884] The server uses the extracted features to train an AI model using a machine learning library, for example, by building and training a Transformer-based model.

[0885] Step 8:

[0886] The trained AI model is tested and evaluated, and adjustments are made as needed. The server manages the versioning of the generated model and stores it securely.

[0887] Step 9:

[0888] The server then integrates the trained AI model into a user interface (chatbot or voice response system), allowing users to interact with the recreated features of the deceased.

[0889] Step 10:

[0890] Users enter questions or messages through the interface, which are sent to the AI ​​model on the server, which uses the model to generate an appropriate response and sends the answer back to the user.

[0891] Step 11:

[0892] The terminal displays the response received from the server to the user, and the user can continue the conversation by sending another question or message.

[0893] Example 1

[0894] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0895] It is necessary to provide a system that can reproduce the characteristics and communication style of the deceased and converse with them while ensuring the security of personal data. It is also difficult to deal with various forms of personal data, including voice data, and to effectively train an AI model to realize the dialogue.

[0896] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0897] In this invention, the server includes means for a user to upload personal data related to the deceased, means for receiving and storing the uploaded personal data, means for encrypting the personal data and storing it in cloud storage or a database, means for preprocessing the stored personal data and converting it into a unified format, means for extracting features from the preprocessed personal data, means for training an AI model based on the extracted features, means for providing an interface using the trained AI model to allow the user to communicate with the deceased, and means for responding to the user's questions and messages through the provided interface. This makes it possible to extract features from various forms of data about the deceased and effectively train the AI ​​model to realize dialogue while ensuring the security of the personal data.

[0898] "Uploading" is the act of a user sending data from their device to a server.

[0899] "Personal data" refers to all information generated by an individual, including text data and audio data relating to a deceased person.

[0900] "Encryption" is the process of converting data using a specific algorithm to make it unreadable to third parties.

[0901] "Cloud storage" is an external storage service that stores and accesses data over the Internet.

[0902] "Preprocessing" is the process of cleaning and formatting data to make it easier for subsequent processing.

[0903] "Features" are important elements or parameters extracted from data, and are information used to train AI models.

[0904] An "AI model" is a learning algorithmic structure built using artificial intelligence techniques and trained to perform a specific task.

[0905] "Interface" refers to the tools and environments through which users interact with a system, including chatbots and voice response systems.

[0906] "Voice recognition technology" is a technology that analyzes voice data and converts it into text.

[0907] A "Transformer model" is a type of deep learning model used in natural language processing, capable of analyzing the meaning and context of text.

[0908] The present invention is a system that can reproduce the characteristics and patterns of a deceased person and communicate with the deceased. To implement this system, the entire system is constructed based on the following procedure.

[0909] First, users use a dedicated application or web interface to upload personal data related to the deceased, including diary entries, blogs, message history, or audio recordings.

[0910] The server then receives the personal data uploaded by the user, encrypts the data, and stores it in cloud storage or a database. The received data is encrypted using the Advanced Encryption Standard (AES) to ensure security. The encryption key is stored in a separate, secure location.

[0911] The server preprocesses the stored data. For text data, unnecessary symbols and HTML tags are removed and the data is normalized. For audio data, the data is converted to text using Google Cloud Speech-to-Text or other speech recognition APIs.

[0912] After preprocessing, the server extracts features from the data. For example, natural language processing techniques such as TF-IDF (Term Frequency-Inverse Document Frequency) and Word2Vec are used to extract important keywords and unique phrases from the text. In the case of audio data, features such as speech rhythm, intonation, and specific pronunciation patterns are also extracted.

[0913] After the features are extracted, the server trains an AI model based on them. It uses machine learning libraries such as TensorFlow and PyTorch to build a Transformer-based model, using the extracted features as training data. Through an iterative process, it learns the deceased person's unique speaking style and knowledge.

[0914] Using the trained AI model, the server provides a user-accessible interface, consisting of a chatbot and a voice response system, allowing users to enjoy virtual conversations with the deceased through a dedicated chat application or web interface.

[0915] As a specific example of operation, a user opens the application, selects and uploads the diary file of the deceased person. The server then encrypts, stores, and preprocesses the file, extracts features, and trains the AI ​​model. Finally, when the user asks through the application, "What's the weather like today?", the AI ​​responds, "It's sunny today," in the style of the deceased person.

[0916] This system can accurately reproduce the characteristics and conversational style of the deceased and enable dialogue with the user, thereby passing on the knowledge and skills of the deceased to the next generation and contributing to the preservation of culture and the continuation of industry.

[0917] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0918] Step 1: Upload user data

[0919] The user opens a dedicated application or web interface, selects a data file (e.g., diary, audio file) related to the deceased, and clicks the upload button. The input data is the personal data selected by the user, and the output is to send the selected file to the server. Specifically, the user clicks the file selection button in the application, selects "diary.txt," and presses the upload button.

[0920] Step 2: Receiving and Encrypting Data

[0921] The server stores the received personal data in a temporary folder. It then encrypts the received data using the AES (Advanced Encryption Standard) algorithm. The input data is the raw personal data sent by the user, and the output is the encrypted data. Specifically, the server receives "diary.txt," encrypts it using the AES algorithm, and saves it in an AWS S3 bucket as "encrypted_diary.txt."

[0922] Step 3: Preprocessing the data

[0923] The server loads the stored data and begins preprocessing. The input data is encrypted personal data, and the output is cleaned and normalized data. The processing involves removing unnecessary symbols and HTML tags from text data, and converting audio data to text using Google Cloud Speech-to-Text. Specifically, it loads "encrypted_diary.txt," decrypts it, and then removes HTML tags and special characters.

[0924] Step 4: Feature extraction

[0925] The server extracts features from the data after preprocessing. The input data is cleaned personal data, and the output is the extracted features. Specific techniques include using natural language processing techniques (e.g., TF-IDF, Word2Vec) to extract important keywords and distinctive phrases from the text. In the case of audio data, features of audio patterns are extracted. Specific operations include extracting feature keywords from "diary_clean.txt" using TF-IDF and saving them in "keyword_list.txt."

[0926] Step 5: Training the AI ​​model

[0927] The server builds an AI model based on the features and trains it. The input data are the extracted features, and the output is a trained AI model. Specifically, a Transformer-based model is created using TensorFlow or PyTorch, and specific features are used as training data. Specifically, a Transformer model is built using TensorFlow, and the extracted "keyword_list.txt" is used as training data to train the model.

[0928] Step 6: Providing an Interface

[0929] The server uses the trained AI model to provide an interface that users can access. The input data is input from the user (text or voice), and the output is the response from the AI ​​model. Specifically, the trained model is integrated into a Flask web application to provide a chat interface. Users can interact with the AI ​​through a dedicated chat application or web interface.

[0930] Step 7: Communicate with users

[0931] The user inputs questions or messages through the interface, and the server responds using a trained AI model. The input data is the user's question or message, and the output is the response from the AI ​​model. Specifically, when a user types "What's the weather like today?" into a chat application, the server responds "It's sunny today" in the style of the deceased.

[0932] (Application example 1)

[0933] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0934] Currently, there are limited ways to reminisce about memories with deceased family and friends. Traditional photo albums and videos are merely static records and do not allow for interactive communication with the deceased. While there are technologies that utilize personal data to recreate the characteristics and speaking style of the deceased, there is a lack of effective integration of advanced natural language processing and voice reproduction technologies.

[0935] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0936] In this invention, the server includes: means for a user to upload personal data related to the deceased; means for receiving and storing the uploaded personal data; means for extracting features from the stored personal data; means for training a generative AI model based on the extracted features; means for providing an interface that allows the user to communicate with the deceased using the trained AI model; means for reproducing the voice and conversation style of the deceased based on the responses of the generative AI model; means for inputting prompts that the user can use to enjoy a conversation with the deceased; and means for performing preprocessing to convert voice data into text data. This enables two-way virtual communication with the deceased, allowing the user to reminisce about their memories in a more realistic and moving way.

[0937] "Means by which users can upload personal data relating to a deceased person" refers to devices or software that allow text and audio data relating to a deceased person to be sent digitally to a server.

[0938] "Means for receiving and storing uploaded personal data" refers to the technology or system that receives submitted personal data and stores it in a secure location.

[0939] "Means for extracting features from stored personal data" refers to algorithms or processes that analyze stored data and identify the characteristics and speech patterns of the deceased.

[0940] "Means for training a generative AI model" refers to the process of feeding data to an AI model using the extracted features to train the model to have the ability to reproduce the speech pattern and characteristics of the deceased.

[0941] "Means for providing an interface" means a user interface that allows a user to interact with an AI model, such as a chat application or a voice response system.

[0942] "Means of recreating the voice and speaking style of the deceased based on the response of a generative AI model" refers to technologies and systems that synthesize the voice and distinctive speaking style of the deceased based on the output of an AI model.

[0943] "Means for inputting prompt text" refers to a device or interface that allows a user to input text to ask or speak to the deceased.

[0944] The "means for performing preprocessing to convert voice data into text data" refers to a technology for generating text from a voice file, such as a voice recognition system.

[0945] This invention is a system that allows users to upload personal data related to the deceased, enabling communication that recreates the characteristics and patterns of the deceased.

[0946] 1. Program Generation

[0947] The user uses their device to upload text and audio data related to the deceased to the server. The server receives this data and stores it in a secure location. The server also extracts features from the stored data and analyzes the deceased's unique speaking style and expressions. Based on these features, the server trains a generative AI model to build a model that mimics the characteristics of the deceased. The server then uses the trained AI model to provide an interface that allows the user to interact with the deceased.

[0948] 2. Explanation of program processing

[0949] This system uses the following major hardware and software:

[0950] Server: Used for data storage and computation. A good example is Amazon EC2.

[0951] Generative AI model: Uses OpenAI API to recreate the speaking style of the deceased.

[0952] Web framework: Django is used to provide the user interface.

[0953] Data processing and calculation:

[0954] The server receives and stores the uploaded personal data. It then preprocesses the stored data and converts it into a unified format. For example, it uses a speech-to-text API to convert voice data into text data. It then uses natural language processing technology to extract features and trains a generative AI model. Once trained, the AI ​​model can reproduce the deceased person's unique speaking style.

[0955] 3. Specific Examples

[0956] Below are some specific scenarios:

[0957] 1. The user opens the application and uploads an audio file of the deceased person.

[0958] 2. The server receives the audio file and stores it in secure cloud storage.

[0959] 3. Preprocess the audio file and convert it to text, for example, using the Google Speech-to-Text API.

[0960] 4. The server analyzes the text data and extracts keywords and phrases characteristic of the deceased.

[0961] 5. Based on the extracted features, a generative AI model is trained using the OpenAI API.

[0962] 6. Using the learning model, users can enjoy interacting with their deceased loved ones through chat applications.

[0963] 7. When the user types "Hello, Grandpa," the system responds by recreating the deceased's distinctive speaking style, "Hello, how are you?"

[0964] Prompt Sentence Examples

[0965] "Grandpa, how was your day today?"

[0966] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0967] Step 1: The user uses a device to upload text and audio data related to the deceased person to the server.

[0968] Input: Text data and audio data

[0969] Specific operation: The user uses the application's upload function to select and send a note or audio file of the deceased person, which is then received by the server.

[0970] Step 2: The server receives the uploaded personal data and stores it in a secure location.

[0971] Input: Text and audio data uploaded by users

[0972] Output: Stored personal data

[0973] What it does: Encrypts received data and stores it in cloud storage. Uses security protocols to ensure data confidentiality.

[0974] Step 3: The server preprocesses the stored data and converts it into a unified format.

[0975] Input: Stored personal data

[0976] Output: Formatted text data

[0977] Specific operations: For text data, remove unnecessary tags and symbols. For audio data, convert audio to text using the Speech-to-Text API.

[0978] Step 4: The server extracts features from the preprocessed data.

[0979] Input: Formatted text data

[0980] Output: Extracted features

[0981] What it does: It uses natural language processing technology to analyze keywords and distinctive phrases in texts to identify patterns in the speech and writing style of the deceased.

[0982] Step 5: The server trains a generative AI model based on the extracted features.

[0983] Input: Extracted features

[0984] Output: A trained generative AI model

[0985] What it does: It uses a machine learning library (e.g., Transformer) to train an AI model, which iterates to learn the deceased person's speaking style.

[0986] Step 6: The server uses the trained AI model to provide an interface that allows the user to interact with the deceased.

[0987] Input: A trained generative AI model

[0988] Output: User interface

[0989] What it does: Build an interface that acts as a chatbot or voice response system and provides it to users in an easily accessible way.

[0990] Step 7: The user uses the terminal to enter a prompt and interact with the deceased.

[0991] Input: prompt statement

[0992] Output: Response text or audio

[0993] How it works: When a user enters a prompt phrase, such as "Hello, Grandpa," the server uses a generative AI model to generate a response in the style of the deceased person, and returns it as text or voice.

[0994] Through these steps, the system can accurately reproduce the characteristics and conversation style of the deceased person and engage in a virtual dialogue with the user.

[0995] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0996] The present invention includes a system that reproduces the characteristics and patterns of the deceased and enables communication between the deceased and the user, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. This system is realized mainly through the process of providing personal data by the user, data processing by the server, learning of an AI model, and emotion recognition and response generation.

[0997] User operations

[0998] Using a dedicated application or web interface, users select and upload personal data (e.g., text files, audio files) related to the deceased, such as diary entries, blogs, message histories, or audio recordings written by the deceased.

[0999] Receiving and storing data

[1000] The server receives the personal data uploaded by the user, encrypts the data immediately after receiving it, and stores it securely in a database or cloud storage, ensuring the confidentiality and integrity of the data.

[1001] Data Preprocessing

[1002] The server performs preprocessing on the stored data. This preprocessing includes removing unnecessary spaces and symbols in the case of text data, and converting audio data into text using a speech recognition API. This converts the data into a unified format, allowing for smooth subsequent processing.

[1003] Feature extraction

[1004] After preprocessing is complete, the server uses natural language processing technology to extract features from the data. Specifically, it analyzes keywords, phrases, and grammatical structures in the text to identify the deceased's writing style and speaking patterns. In the case of audio data, it also extracts the rhythm and intonation of speech, as well as specific pronunciation patterns.

[1005] Model training

[1006] After the features are extracted, the server trains an AI model based on the extracted features. For example, a Transformer-based model is built and trained using a machine learning library. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge. This trained AI model is then used to interact with the user.

[1007] Emotion Recognition and Response Generation

[1008] The emotion engine recognizes the user's emotions in response to input from the user. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the AI ​​model and reflected when generating responses. This allows for the generation of appropriate responses according to the user's emotional state, resulting in more human-like interactions.

[1009] Providing an interface

[1010] The server provides users with an interface (e.g., a chatbot or voice response system) that integrates the trained AI model and emotion engine. Through this interface, users can enjoy conversations based on the recreated features and emotion recognition of the deceased person.

[1011] Specific examples

[1012] 1. User Data Provision:

[1013] The user opens the application, selects and uploads the diary and audio files of the deceased person.

[1014] 2. Receiving and storing data:

[1015] The server receives the deceased person's diary files and audio files, encrypts the data, and stores it in cloud storage.

[1016] 3. Data preprocessing:

[1017] The server cleans the diary text, removing unnecessary spaces and symbols, and converts the audio file into text using a speech recognition API.

[1018] 4. Feature extraction:

[1019] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[1020] 5. Train the model:

[1021] The server uses a machine learning library to train a Transformer model based on the features.

[1022] 6. Emotion Recognition and Response Generation:

[1023] In response to text or voice input from the user, the server uses an emotion engine to identify the user's emotion, and the AI ​​model generates the optimal response based on that emotion.

[1024] 7. Providing an interface:

[1025] The server provides a chatbot that integrates a trained model and an emotion engine, making it accessible to users.

[1026] 8. Communication:

[1027] The user types, "How are you feeling today?" and the server's AI model recognizes the user's emotion of joy and replies, "I feel great today," in the style of the deceased.

[1028] In this way, the system can not only accurately reproduce the characteristics and conversational style of the deceased, but also identify the user's emotions and provide appropriate responses based on those emotions, enabling richer communication.

[1029] The processing flow will be explained below.

[1030] Step 1:

[1031] The user selects and uploads personal data related to the deceased (e.g., text files, audio files) through a dedicated application or web interface.

[1032] Step 2:

[1033] The terminal transmits the data selected by the user to the server, and before transmitting the data, it is encrypted as necessary.

[1034] Step 3:

[1035] The server receives the data sent from the device, then immediately encrypts it again and stores it securely in a database or cloud storage.

[1036] Step 4:

[1037] The server starts preprocessing the stored data. For text data, unnecessary spaces and special characters are deleted and the format is standardized. For audio data, a speech recognition API is used to convert it into text data.

[1038] Step 5:

[1039] The server uses natural language processing technology to extract features from the preprocessed data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the characteristics of the deceased. In the case of audio data, it also extracts the rhythm and intonation of speech, as well as specific pronunciation patterns.

[1040] Step 6:

[1041] The server uses the extracted features to train an AI model using a machine learning library, such as building and training a Transformer-based model. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge.

[1042] Step 7:

[1043] The server uses an emotion engine to identify the emotion of user input (text and voice). The emotion engine performs text and voice analysis to determine emotional states such as joy, sadness, and anger.

[1044] Step 8:

[1045] The server adjusts the AI ​​model's response based on the emotional information obtained from the emotion engine. By incorporating the emotional information into response generation, an appropriate response is generated according to the user's emotional state.

[1046] Step 9:

[1047] The server provides an interface that integrates trained AI models and emotion engines. For example, users can interact with AI through chatbots or voice response systems.

[1048] Step 10:

[1049] When a user enters a question or message through the interface, it is sent to the server's AI model and emotion engine. The server uses the emotion engine to identify the user's emotion and uses the AI ​​model to generate an appropriate response. This response is then sent back to the user.

[1050] Step 11:

[1051] The device displays the response received from the server to the user. The user can also send questions or messages again, allowing for a continuous dialogue. This allows the user to experience a more human-like dialogue through an AI model that recognizes emotions and recreates the characteristics of the deceased.

[1052] Example 2

[1053] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1054] A system that enables conversation with the deceased must accurately reproduce the characteristics and conversational style of the deceased, recognize the user's emotions, and reflect them in the conversation. It must also convert various data formats into a unified format and efficiently extract features while ensuring the confidentiality and integrity of personal data.

[1055] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1056] In this invention, the server includes: a means for a user to upload personal data related to the deceased; a means for receiving and saving the uploaded personal data; a means for preprocessing the saved personal data, such as deleting unnecessary spaces and symbols in the case of text data and converting audio data into text using speech recognition technology; a means for extracting features from the preprocessed data; a means for training an AI model based on the extracted features to reproduce the deceased's unique conversational style; a means for recognizing emotions in response to user input and generating appropriate responses according to those emotions; and a means for providing an interface that allows the user to communicate with the deceased using the trained AI model and emotion engine. This makes it possible to reproduce the deceased's characteristics and conversational style while generating responses according to the user's emotions.

[1057] A "user" is a person who wishes to use the system to interact with the deceased.

[1058] A "deceased person" is someone who has already passed away, and the system will recreate that person's characteristics and conversation style.

[1059] "Personal data" is information related to a deceased person that is uploaded by a user, including text files, audio files, diaries, message history, etc.

[1060] "Means for uploading" refers to the method by which a user provides personal data relating to the deceased person to the system, including a dedicated application or web interface.

[1061] "Means of receiving and storing" refers to the process by which the server receives the uploaded personal data and stores it safely.

[1062] "Preprocessing" refers to processing performed on stored personal data, and includes deleting unnecessary spaces and symbols in the case of text data, and converting audio data into text using voice recognition technology.

[1063] "Features" are useful information extracted from personal data, and include keywords, phrases, grammatical structures, speech rhythm and intonation, etc.

[1064] An "AI model" is a mathematical model built and trained using machine learning techniques to recreate the speaking style and characteristics of a deceased person.

[1065] An "emotion engine" is a technology that recognizes emotions in response to user input and generates a response that corresponds to that emotion.

[1066] An "interface" is an interactive environment that allows users to communicate with the deceased using trained AI models and emotion engines, including chatbots and voice response systems.

[1067] A "uniform format" is a state in which data has been converted into a common format through preprocessing, allowing subsequent processing to proceed smoothly.

[1068] The present invention includes a system that reproduces the characteristics and patterns of the deceased and enables communication between the deceased and the user, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. This system is realized mainly through the process of providing personal data by the user, data processing by the server, learning of an AI model, and emotion recognition and response generation.

[1069] System configuration

[1070] 1. User Interface

[1071] Users upload personal data relating to the deceased using a dedicated application or web interface, which is designed to be easy for users to navigate and guide them through the selection and upload process.

[1072] 2. Receipt and storage of data

[1073] The server receives personal data uploaded by users, encrypts the data immediately after receiving it, and stores it securely in a database or cloud storage, thus ensuring the confidentiality and integrity of the received data.

[1074] 3. Data Preprocessing

[1075] The server performs preprocessing on the stored data. For text data, unnecessary spaces and symbols are removed, and for audio data, speech recognition technology (e.g., Google Speech-to-Text API) is used to convert it into text. This preprocessing converts the data into a unified format, allowing for smooth subsequent processing.

[1076] 4. Feature Extraction

[1077] The server extracts features from the pre-processed data using natural language processing technology (e.g., spaCy). It analyzes keywords, phrases, and grammatical structures in the text to identify the deceased's style and speaking patterns. In the case of audio data, speech rhythm, intonation, and specific pronunciation patterns are also extracted.

[1078] 5. Training the Model

[1079] The server trains an AI model based on the extracted features. For example, it builds and trains a Transformer-based model using a machine learning library (e.g., TensorFlow, PyTorch). During the training process, iterative processing is performed on the provided data, and the model acquires the ability to reproduce the deceased's unique speaking style and knowledge.

[1080] 6. Emotion Recognition and Response Generation

[1081] The emotion engine recognizes the user's emotions in response to input from the user. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the AI ​​model and reflected when generating responses. This generates appropriate responses according to the user's emotional state, resulting in more human-like interactions.

[1082] 7. Providing an Interface

[1083] The server provides an interface that integrates a trained AI model and an emotion engine. Through this interface (e.g., a chatbot or voice response system), users can enjoy conversations based on the recreated features and emotion recognition of the deceased.

[1084] Specific examples

[1085] 1. User Data Provision:

[1086] Users open the application, select and upload the deceased person's diary and audio files.

[1087] 2. Receiving and storing data:

[1088] The server receives the deceased person's diary files and audio files, encrypts the data, and stores it in cloud storage.

[1089] 3. Data preprocessing:

[1090] The server cleans the diary text, removing unnecessary spaces and symbols, and converts the audio file to text using the Google Speech-to-Text API.

[1091] 4. Feature extraction:

[1092] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[1093] 5. Train the model:

[1094] The server uses TensorFlow to train a Transformer model based on the features.

[1095] 6. Emotion Recognition and Response Generation:

[1096] In response to text or voice input from the user, the server uses an emotion engine to identify the user's emotion, and the AI ​​model generates the optimal response based on that emotion.

[1097] 7. Providing an interface:

[1098] The server provides a chatbot that integrates a trained model and an emotion engine, making it accessible to users.

[1099] 8. Communication:

[1100] The user types, "How are you feeling today?" and the server's AI model recognizes the user's emotion of joy and replies, "I feel great today," in the style of the deceased.

[1101] This system can not only reproduce the characteristics and conversational style of the deceased, but also identify the user's emotions and generate appropriate responses based on those emotions, enabling richer communication.

[1102] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1103] Step 1: Provide user data

[1104] The user opens a dedicated application or web interface and selects and uploads personal data related to the deceased (e.g., diary file "diary.txt" or audio file "voice_recording.mp3").

[1105] Input: User-selected text and audio files

[1106] Output: The uploaded file is sent to the server as input.

[1107] Step 2: Receiving and storing data

[1108] The server receives the personal data uploaded by the user, encrypts it with the AES-256 algorithm, and stores it in cloud storage.

[1109] Input: User-uploaded text and audio files

[1110] Output: The encrypted data is stored in cloud storage.

[1111] Step 3: Preprocessing the data

[1112] The server loads the stored data from cloud storage. For text data, it uses regular expressions to remove unnecessary spaces and symbols. For audio data, it uses the Google Speech-to-Text API to convert the audio to text.

[1113] Input: Encrypted text and audio data stored in cloud storage

[1114] Output: Clean text data and converted text data

[1115] Step 4: Feature extraction

[1116] The server extracts features from the preprocessed data using natural language processing technology (e.g., spaCy). Specifically, it analyzes keywords, phrases, grammatical structures, and speech rhythm and intonation in the text.

[1117] Input: Preprocessed text data

[1118] Output: Extracted keywords, phrases, grammatical structures, speech rhythm, intonation, and other features

[1119] Step 5: Training the model

[1120] The server trains an AI model based on the features. It uses a machine learning library (e.g., TensorFlow, PyTorch) to build and train a Transformer-based model.

[1121] Input: extracted features

[1122] Output: Trained AI model

[1123] Step 6: Emotion recognition and response generation

[1124] The user inputs a message into the system. The emotion engine on the server analyzes the user's input message and identifies the emotional state (e.g., joy, sadness, anger, etc.). The AI ​​model generates an appropriate response based on the emotion.

[1125] Input: Input message from the user

[1126] Output: Response message generated by the AI ​​model according to the emotion

[1127] Step 7: Providing an Interface

[1128] The server provides an interface (e.g., a chatbot) that integrates a trained AI model and an emotion engine. Users can access this interface through a browser or application.

[1129] Input: trained AI model, emotion engine, user message

[1130] Output: The interface presented to the user

[1131] Step 8: Communicate

[1132] The user initiates a dialogue with the deceased, and the server analyzes the user's input and generates a response based on their emotional state. For example, if the user types, "How are you feeling today?", the server's AI model will recognize the emotion of joy and generate the response, "I feel great today."

[1133] Input: User interaction message

[1134] Output: An appropriate response message based on the sentiment

[1135] This allows the characteristics and conversation style of the deceased to be reproduced, enabling more realistic communication that responds to the user's emotions.

[1136] (Application example 2)

[1137] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1138] In current virtual stores, it is difficult to provide personalized product recommendations to users, and there are currently no systems that provide an emotional shopping experience based on personal data related to the deceased. Furthermore, there is a lack of interfaces that can recognize the user's emotions and respond or provide recommendations accordingly. Therefore, there is a need for a system that can recognize the user's emotional state and generate optimal responses based on that, especially for a virtual store that can provide emotionally rich communication through dialogue that reflects the characteristics of the deceased.

[1139] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1140] In this invention, the server includes means for a user to upload individual data related to the deceased, means for receiving and storing the uploaded individual data, means for extracting feature data from the stored individual data, means for training a generative AI model based on the extracted feature data, means for recognizing the user's emotional state using an emotion engine, means for providing an interface that allows the user to communicate with the deceased in an emotionally adapted manner using the trained generative AI model and the emotion engine, and means for making product suggestions through dialogue in a virtual store that reflects the characteristics of the deceased. This allows the user to receive personalized product suggestions through emotion-based communication with the deceased.

[1141] "User" refers to an individual who uses the system to upload personal data related to the deceased and communicate in the virtual store.

[1142] A "deceased person" refers to an entity that had a past relationship with a user of the system, and is recreated by providing a basis for individual data related to the deceased.

[1143] "Individual data" is a general term for digital information that includes unique information related to the deceased, such as diaries, message history, and audio files.

[1144] "Uploading means" refers to the hardware and software that allows users to transfer individual data to the server.

[1145] "Means for receiving and storing" refers to the function for receiving uploaded individual data on the server side and storing it safely.

[1146] "Characteristic data" is data extracted from the stored individual data that indicates the unique characteristics of the deceased, such as their sentence structure and pronunciation patterns.

[1147] "Learning means" refers to a method and apparatus for training a generative AI model based on extracted feature data to reproduce the features of the deceased.

[1148] "Generative AI model" refers to an artificial intelligence model that is trained using machine learning algorithms to reproduce the conversational style and responses of a deceased person.

[1149] "Emotion Engine" refers to algorithms and software for recognizing a user's emotional state and generating responses based on that.

[1150] "Interface" refers to the user interaction environment that allows users to communicate with the deceased using the generative AI model and emotion engine.

[1151] A "virtual store" refers to a store-like environment that exists in a virtual space, and is a virtual platform where users can receive product suggestions and make purchases.

[1152] "Product suggestion means" refers to a method and device for reflecting the characteristics of the deceased person in a virtual store and suggesting appropriate products to the user.

[1153] This invention includes a system that reproduces the characteristics and patterns of the deceased and allows the user to communicate with the deceased, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. The system's primary purpose is to suggest products through dialogue that reflects the characteristics of the deceased in a virtual store.

[1154] Generating a Program

[1155] This system is realized based on the following program configuration.

[1156] 1. User data provision:

[1157] Using a dedicated application, users upload personal data related to the deceased, such as diary entries, message history, or audio recordings.

[1158] 2. Receiving and storing data:

[1159] The server receives the individual data uploaded by users and securely stores it in a database or cloud storage. The data is encrypted immediately after receiving it to ensure confidentiality and integrity of the data.

[1160] 3. Data preprocessing:

[1161] The server performs preprocessing on the stored data, such as deleting unnecessary spaces and symbols in the case of text data, or converting audio data into text using a speech recognition API, thereby converting the data into a unified format.

[1162] 4. Extracting feature data:

[1163] After the preprocessing is complete, the server uses natural language processing technology to extract feature data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the deceased's writing style and speaking patterns.

[1164] 5. Training the generative AI model:

[1165] After the feature data is extracted, the server trains a generative AI model based on the extracted feature data. For example, it uses a machine learning library to build and train a Transformer-based model. During the training process, iterative processing is performed on the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge.

[1166] 6. Emotion Recognition and Response Generation:

[1167] The emotion engine recognizes the user's emotions in response to user input. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the generative AI model and reflected when generating responses. This allows for the generation of appropriate responses according to the user's emotional state.

[1168] 7. Virtual store interface:

[1169] The server provides an interface that integrates a trained generative AI model and an emotion engine. Through smart glasses, users can enjoy conversations based on the deceased's recreated features and emotion recognition within a virtual store. For example, if a user types, "What do you think of this red dress?", the server's AI model will recognize the user's emotion and reply in the deceased's style, "This dress is vibrant and lovely. I think it suits you well."

[1170] Hardware and software used

[1171] Hardware: Server, smart glasses

[1172] Software: Machine learning libraries (e.g., Transformers), speech recognition APIs, databases and cloud storage, emotion engines

[1173] Specific examples

[1174] Scenario: A user wears smart glasses and enjoys shopping while walking through a virtual store.

[1175] Example dialogue:

[1176] User: "What do you think about this red dress?"

[1177] Virtual Guide: "This dress is vibrant and beautiful. I think it would look great on you."

[1178] Prompt Sentence Examples

[1179] The following text recognizes the user's emotional state when they suggest a product and generates a response based on that emotion, continuing the dialogue while reflecting the characteristics and style of the deceased.

[1180] User: What do you think about this red dress?

[1181] Emotional state: Joy

[1182] Response: This dress is vibrant and beautiful. I think it looks great on you.

[1183] In this way, the system not only reproduces the characteristics and conversational style of the deceased with high accuracy, but also identifies the user's emotions and returns appropriate responses based on those emotions, enabling richer communication.

[1184] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1185] Step 1:

[1186] The user uploads personal data relating to the deceased person using the application.

[1187] Specifically, the user selects data such as diary entries, message history, and audio files and uploads them on the application. At this time, the application generates an API request to transfer the data to the server and sends the data.

[1188] Input: Individual data such as diary entries, message history, and audio files

[1189] Output: Data transfer to the server in the form of an API request

[1190] Step 2:

[1191] The server receives the uploaded individual data, securely encrypts it, and stores it in a database or cloud storage.

[1192] Specifically, the server immediately encrypts the data it receives and stores it in a database or cloud storage, ensuring the confidentiality and integrity of the data.

[1193] Input: Individual data transferred by the user

[1194] Output: Encrypted database or cloud storage

[1195] Step 3:

[1196] The server performs preprocessing on the stored individual data.

[1197] Specifically, in the case of text data, unnecessary spaces and symbols are removed, and in the case of voice data, it is converted into text data using a voice recognition API. By converting the data into a unified format, subsequent processing becomes smoother.

[1198] Input: Data from an encrypted database or cloud storage

[1199] Output: Preprocessed data in a unified format

[1200] Step 4:

[1201] The server extracts feature data from the preprocessed data.

[1202] Specifically, natural language processing techniques are used to analyze keywords, phrases, and grammatical structures in text to identify the deceased's writing style and speaking patterns, and in the case of audio data, to extract speech rhythm, intonation, and specific pronunciation patterns.

[1203] Input: Data converted into a unified format

[1204] Output: characteristic data of the deceased

[1205] Step 5:

[1206] The server trains a generative AI model based on the extracted feature data.

[1207] Specifically, a Transformer-based AI model is built and trained using machine learning libraries (e.g., Transformers). During the training process, the model iterates on the provided data to gain the ability to reproduce the deceased person's unique speaking style and knowledge.

[1208] Input: Feature data

[1209] Output: A trained generative AI model

[1210] Step 6:

[1211] The user provides input through an interface to interact with the generative AI model.

[1212] Specifically, using smart glasses in a virtual store, users can input questions or comments by voice or text, and this data is sent to a server.

[1213] Input: User questions and comments

[1214] Output: Transfer of input data to the server

[1215] Step 7:

[1216] The server uses an emotion engine to recognize the user's emotional state and generates an appropriate response using a generative AI model.

[1217] Specifically, the emotion engine identifies the emotional state (e.g., joy, sadness, etc.) from the user's input data (voice or text) and feeds that emotion back to the generative AI model, which then generates the optimal response by taking into account the characteristics of the deceased and the user's emotions.

[1218] Input: User input data, emotion engine recognition results

[1219] Output: An appropriate response based on the user's sentiment

[1220] Step 8:

[1221] The server provides the generated response to the user in the virtual store.

[1222] Specifically, the generated response is communicated to the user through the smart glasses. For example, if a user asks, "What do you think of this red dress?", the server's AI model will generate a response such as, "This dress is vibrant and beautiful. I think it suits you well," and present it to the user through the smart glasses.

[1223] Input: The generated response

[1224] Output: Presentation of response through smart glasses

[1225] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1226] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1227] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1228] [Fourth embodiment]

[1229] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1230] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1231] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1232] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1233] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1234] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1235] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1236] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1237] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1238] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1239] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1240] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1241] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1242] This invention is a system that can reproduce the characteristics and patterns of a deceased person and communicate with them. This system is realized mainly by training an AI model on a server based on personal data provided by the user.

[1243] User Actions

[1244] Using a dedicated application or web interface, users upload personal data (e.g., text data, audio data) related to the deceased, such as diary entries, blogs, message histories, or audio recordings written by the deceased.

[1245] Receiving and storing data

[1246] The server receives the personal data uploaded by the user, which is then encrypted for security purposes and stored in a database or cloud storage, ensuring the confidentiality of the data.

[1247] Data Preprocessing

[1248] The server performs preprocessing on the stored data. This preprocessing includes removing unnecessary symbols and tags from text data, and converting audio data into text using a speech recognition API. This converts the data into a unified format, allowing for smooth subsequent processing.

[1249] Feature extraction

[1250] After preprocessing, the server extracts features from the data. Specifically, it uses natural language processing technology to analyze keywords, distinctive phrases, and grammatical structures in the text to identify the deceased's writing style and speaking patterns. In the case of audio data, the server also extracts speech rhythm, intonation, and specific pronunciation patterns.

[1251] Model training

[1252] After the features are extracted, the server trains an AI model based on the extracted features. For example, a Transformer-based model is built and trained using a machine learning library. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge. This trained AI model is then used to interact with the user.

[1253] Providing an interface

[1254] Once the trained AI model is complete, the server integrates it into a user-accessible interface (e.g., a chatbot or voice response system). This interface allows users to easily access and enjoy virtual conversations with the deceased. A dedicated chat application or web interface is provided, through which users can ask questions and enjoy conversations with the AI.

[1255] Specific examples

[1256] 1. User data provision:

[1257] The user opens the application, selects and uploads the diary file of the deceased person.

[1258] 2. Receiving and storing data:

[1259] The server receives the diary files of the deceased person, encrypts the data, and stores it in cloud storage.

[1260] 3. Data preprocessing:

[1261] The server cleans the diary text and removes unnecessary HTML tags.

[1262] 4. Feature extraction:

[1263] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[1264] 5. Train the model:

[1265] The server uses a machine learning library to train a Transformer model based on the features.

[1266] 6. Providing an interface:

[1267] The server integrates the trained model into the application and makes the chatbot available for users to access.

[1268] 7. Communication:

[1269] Through the application, the user types, "What's the weather like today?" and the AI ​​responds, "It's sunny today," in the style of the deceased.

[1270] In this way, the system can accurately reproduce the characteristics and conversational style of the deceased and engage in dialogue with the user, thereby enabling the knowledge and skills of the deceased to be passed on to future generations, contributing to the preservation of culture and the continuation of industry.

[1271] The processing flow will be explained below.

[1272] Step 1:

[1273] The user selects and uploads personal data related to the deceased (e.g., text files, audio files) through a dedicated application or web interface.

[1274] Step 2:

[1275] The terminal transmits the data selected by the user to the server, and before transmitting the data, it is encrypted as necessary.

[1276] Step 3:

[1277] The server receives the data sent from the device, and immediately after receiving it, it re-encrypts the data and stores it securely in a database or cloud storage.

[1278] Step 4:

[1279] The server starts preprocessing the stored data. In the case of text data, unnecessary spaces and special characters are removed and the format is standardized.

[1280] Step 5:

[1281] If voice data is included, the server converts the voice data into text data using a voice recognition API, and this converted text data also undergoes preprocessing.

[1282] Step 6:

[1283] After preprocessing is complete, the server uses natural language processing technology to extract features from the data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the characteristics of the deceased.

[1284] Step 7:

[1285] The server uses the extracted features to train an AI model using a machine learning library, for example, by building and training a Transformer-based model.

[1286] Step 8:

[1287] The trained AI model is tested and evaluated, and adjustments are made as needed. The server manages the versioning of the generated model and stores it securely.

[1288] Step 9:

[1289] The server then integrates the trained AI model into a user interface (chatbot or voice response system), allowing users to interact with the recreated features of the deceased.

[1290] Step 10:

[1291] Users enter questions or messages through the interface, which are sent to the AI ​​model on the server, which uses the model to generate an appropriate response and sends the answer back to the user.

[1292] Step 11:

[1293] The terminal displays the response received from the server to the user, and the user can continue the conversation by sending another question or message.

[1294] Example 1

[1295] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1296] It is necessary to provide a system that can reproduce the characteristics and communication style of the deceased and converse with them while ensuring the security of personal data. It is also difficult to deal with various forms of personal data, including voice data, and to effectively train an AI model to realize the dialogue.

[1297] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1298] In this invention, the server includes means for a user to upload personal data related to the deceased, means for receiving and storing the uploaded personal data, means for encrypting the personal data and storing it in cloud storage or a database, means for preprocessing the stored personal data and converting it into a unified format, means for extracting features from the preprocessed personal data, means for training an AI model based on the extracted features, means for providing an interface using the trained AI model to allow the user to communicate with the deceased, and means for responding to the user's questions and messages through the provided interface. This makes it possible to extract features from various forms of data about the deceased and effectively train the AI ​​model to realize dialogue while ensuring the security of the personal data.

[1299] "Uploading" is the act of a user sending data from their device to a server.

[1300] "Personal data" refers to all information generated by an individual, including text data and audio data relating to a deceased person.

[1301] "Encryption" is the process of converting data using a specific algorithm to make it unreadable to third parties.

[1302] "Cloud storage" is an external storage service that stores and accesses data over the Internet.

[1303] "Preprocessing" is the process of cleaning and formatting data to make it easier for subsequent processing.

[1304] "Features" are important elements or parameters extracted from data, and are information used to train AI models.

[1305] An "AI model" is a learning algorithmic structure built using artificial intelligence techniques and trained to perform a specific task.

[1306] "Interface" refers to the tools and environments through which users interact with a system, including chatbots and voice response systems.

[1307] "Voice recognition technology" is a technology that analyzes voice data and converts it into text.

[1308] A "Transformer model" is a type of deep learning model used in natural language processing, capable of analyzing the meaning and context of text.

[1309] The present invention is a system that can reproduce the characteristics and patterns of a deceased person and communicate with the deceased. To implement this system, the entire system is constructed based on the following procedure.

[1310] First, users use a dedicated application or web interface to upload personal data related to the deceased, including diary entries, blogs, message history, or audio recordings.

[1311] The server then receives the personal data uploaded by the user, encrypts the data, and stores it in cloud storage or a database. The received data is encrypted using the Advanced Encryption Standard (AES) to ensure security. The encryption key is stored in a separate, secure location.

[1312] The server preprocesses the stored data. For text data, unnecessary symbols and HTML tags are removed and the data is normalized. For audio data, the data is converted to text using Google Cloud Speech-to-Text or other speech recognition APIs.

[1313] After preprocessing, the server extracts features from the data. For example, natural language processing techniques such as TF-IDF (Term Frequency-Inverse Document Frequency) and Word2Vec are used to extract important keywords and unique phrases from the text. In the case of audio data, features such as speech rhythm, intonation, and specific pronunciation patterns are also extracted.

[1314] After the features are extracted, the server trains an AI model based on them. It uses machine learning libraries such as TensorFlow and PyTorch to build a Transformer-based model, using the extracted features as training data. Through an iterative process, it learns the deceased person's unique speaking style and knowledge.

[1315] Using the trained AI model, the server provides a user-accessible interface, consisting of a chatbot and a voice response system, allowing users to enjoy virtual conversations with the deceased through a dedicated chat application or web interface.

[1316] As a specific example of operation, a user opens the application, selects and uploads the diary file of the deceased person. The server then encrypts, stores, and preprocesses the file, extracts features, and trains the AI ​​model. Finally, when the user asks through the application, "What's the weather like today?", the AI ​​responds, "It's sunny today," in the style of the deceased person.

[1317] This system can accurately reproduce the characteristics and conversational style of the deceased and enable dialogue with the user, thereby passing on the knowledge and skills of the deceased to the next generation and contributing to the preservation of culture and the continuation of industry.

[1318] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1319] Step 1: Upload user data

[1320] The user opens a dedicated application or web interface, selects a data file (e.g., diary, audio file) related to the deceased, and clicks the upload button. The input data is the personal data selected by the user, and the output is to send the selected file to the server. Specifically, the user clicks the file selection button in the application, selects "diary.txt," and presses the upload button.

[1321] Step 2: Receiving and Encrypting Data

[1322] The server stores the received personal data in a temporary folder. It then encrypts the received data using the AES (Advanced Encryption Standard) algorithm. The input data is the raw personal data sent by the user, and the output is the encrypted data. Specifically, the server receives "diary.txt," encrypts it using the AES algorithm, and saves it in an AWS S3 bucket as "encrypted_diary.txt."

[1323] Step 3: Preprocessing the data

[1324] The server loads the stored data and begins preprocessing. The input data is encrypted personal data, and the output is cleaned and normalized data. The processing involves removing unnecessary symbols and HTML tags from text data, and converting audio data to text using Google Cloud Speech-to-Text. Specifically, it loads "encrypted_diary.txt," decrypts it, and then removes HTML tags and special characters.

[1325] Step 4: Feature extraction

[1326] The server extracts features from the data after preprocessing. The input data is cleaned personal data, and the output is the extracted features. Specific techniques include using natural language processing techniques (e.g., TF-IDF, Word2Vec) to extract important keywords and distinctive phrases from the text. In the case of audio data, features of audio patterns are extracted. Specific operations include extracting feature keywords from "diary_clean.txt" using TF-IDF and saving them in "keyword_list.txt."

[1327] Step 5: Training the AI ​​model

[1328] The server builds an AI model based on the features and trains it. The input data are the extracted features, and the output is a trained AI model. Specifically, a Transformer-based model is created using TensorFlow or PyTorch, and specific features are used as training data. Specifically, a Transformer model is built using TensorFlow, and the extracted "keyword_list.txt" is used as training data to train the model.

[1329] Step 6: Providing an Interface

[1330] The server uses the trained AI model to provide an interface that users can access. The input data is input from the user (text or voice), and the output is the response from the AI ​​model. Specifically, the trained model is integrated into a Flask web application to provide a chat interface. Users can interact with the AI ​​through a dedicated chat application or web interface.

[1331] Step 7: Communicate with users

[1332] The user inputs questions or messages through the interface, and the server responds using a trained AI model. The input data is the user's question or message, and the output is the response from the AI ​​model. Specifically, when a user types "What's the weather like today?" into a chat application, the server responds "It's sunny today" in the style of the deceased.

[1333] (Application example 1)

[1334] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1335] Currently, there are limited ways to reminisce about memories with deceased family and friends. Traditional photo albums and videos are merely static records and do not allow for interactive communication with the deceased. While there are technologies that utilize personal data to recreate the characteristics and speaking style of the deceased, there is a lack of effective integration of advanced natural language processing and voice reproduction technologies.

[1336] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1337] In this invention, the server includes: means for a user to upload personal data related to the deceased; means for receiving and storing the uploaded personal data; means for extracting features from the stored personal data; means for training a generative AI model based on the extracted features; means for providing an interface that allows the user to communicate with the deceased using the trained AI model; means for reproducing the voice and conversation style of the deceased based on the responses of the generative AI model; means for inputting prompts that the user can use to enjoy a conversation with the deceased; and means for performing preprocessing to convert voice data into text data. This enables two-way virtual communication with the deceased, allowing the user to reminisce about their memories in a more realistic and moving way.

[1338] "Means by which users can upload personal data relating to a deceased person" refers to devices or software that allow text and audio data relating to a deceased person to be sent digitally to a server.

[1339] "Means for receiving and storing uploaded personal data" refers to the technology or system that receives submitted personal data and stores it in a secure location.

[1340] "Means for extracting features from stored personal data" refers to algorithms or processes that analyze stored data and identify the characteristics and speech patterns of the deceased.

[1341] "Means for training a generative AI model" refers to the process of feeding data to an AI model using the extracted features to train the model to have the ability to reproduce the speech pattern and characteristics of the deceased.

[1342] "Means for providing an interface" means a user interface that allows a user to interact with an AI model, such as a chat application or a voice response system.

[1343] "Means of recreating the voice and speaking style of the deceased based on the response of a generative AI model" refers to technologies and systems that synthesize the voice and distinctive speaking style of the deceased based on the output of an AI model.

[1344] "Means for inputting prompt text" refers to a device or interface that allows a user to input text to ask or speak to the deceased.

[1345] The "means for performing preprocessing to convert voice data into text data" refers to a technology for generating text from a voice file, such as a voice recognition system.

[1346] This invention is a system that allows users to upload personal data related to the deceased, enabling communication that recreates the characteristics and patterns of the deceased.

[1347] 1. Program Generation

[1348] The user uses their device to upload text and audio data related to the deceased to the server. The server receives this data and stores it in a secure location. The server also extracts features from the stored data and analyzes the deceased's unique speaking style and expressions. Based on these features, the server trains a generative AI model to build a model that mimics the characteristics of the deceased. The server then uses the trained AI model to provide an interface that allows the user to interact with the deceased.

[1349] 2. Explanation of program processing

[1350] This system uses the following major hardware and software:

[1351] Server: Used for data storage and computation. A good example is Amazon EC2.

[1352] Generative AI model: Uses OpenAI API to recreate the speaking style of the deceased.

[1353] Web framework: Django is used to provide the user interface.

[1354] Data processing and calculation:

[1355] The server receives and stores the uploaded personal data. It then preprocesses the stored data and converts it into a unified format. For example, it uses a speech-to-text API to convert voice data into text data. It then uses natural language processing technology to extract features and trains a generative AI model. Once trained, the AI ​​model can reproduce the deceased person's unique speaking style.

[1356] 3. Specific Examples

[1357] Below are some specific scenarios:

[1358] 1. The user opens the application and uploads an audio file of the deceased person.

[1359] 2. The server receives the audio file and stores it in secure cloud storage.

[1360] 3. Preprocess the audio file and convert it to text, for example, using the Google Speech-to-Text API.

[1361] 4. The server analyzes the text data and extracts keywords and phrases characteristic of the deceased.

[1362] 5. Based on the extracted features, a generative AI model is trained using the OpenAI API.

[1363] 6. Using the learning model, users can enjoy interacting with their deceased loved ones through chat applications.

[1364] 7. When the user types "Hello, Grandpa," the system responds by recreating the deceased's distinctive speaking style, "Hello, how are you?"

[1365] Prompt Sentence Examples

[1366] "Grandpa, how was your day today?"

[1367] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1368] Step 1: The user uses a device to upload text and audio data related to the deceased person to the server.

[1369] Input: Text data and audio data

[1370] Specific operation: The user uses the application's upload function to select and send a note or audio file of the deceased person, which is then received by the server.

[1371] Step 2: The server receives the uploaded personal data and stores it in a secure location.

[1372] Input: Text and audio data uploaded by users

[1373] Output: Stored personal data

[1374] What it does: Encrypts received data and stores it in cloud storage. Uses security protocols to ensure data confidentiality.

[1375] Step 3: The server preprocesses the stored data and converts it into a unified format.

[1376] Input: Stored personal data

[1377] Output: Formatted text data

[1378] Specific operations: For text data, remove unnecessary tags and symbols. For audio data, convert audio to text using the Speech-to-Text API.

[1379] Step 4: The server extracts features from the preprocessed data.

[1380] Input: Formatted text data

[1381] Output: Extracted features

[1382] What it does: It uses natural language processing technology to analyze keywords and distinctive phrases in texts to identify patterns in the speech and writing style of the deceased.

[1383] Step 5: The server trains a generative AI model based on the extracted features.

[1384] Input: Extracted features

[1385] Output: A trained generative AI model

[1386] What it does: It uses a machine learning library (e.g., Transformer) to train an AI model, which iterates to learn the deceased person's speaking style.

[1387] Step 6: The server uses the trained AI model to provide an interface that allows the user to interact with the deceased.

[1388] Input: A trained generative AI model

[1389] Output: User interface

[1390] What it does: Build an interface that acts as a chatbot or voice response system and provides it to users in an easily accessible way.

[1391] Step 7: The user uses the terminal to enter a prompt and interact with the deceased.

[1392] Input: prompt statement

[1393] Output: Response text or audio

[1394] How it works: When a user enters a prompt phrase, such as "Hello, Grandpa," the server uses a generative AI model to generate a response in the style of the deceased person, and returns it as text or voice.

[1395] Through these steps, the system can accurately reproduce the characteristics and conversation style of the deceased person and engage in a virtual dialogue with the user.

[1396] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1397] The present invention includes a system that reproduces the characteristics and patterns of the deceased and enables communication between the deceased and the user, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. This system is realized mainly through the process of providing personal data by the user, data processing by the server, learning of an AI model, and emotion recognition and response generation.

[1398] User operations

[1399] Using a dedicated application or web interface, users select and upload personal data (e.g., text files, audio files) related to the deceased, such as diary entries, blogs, message histories, or audio recordings written by the deceased.

[1400] Receiving and storing data

[1401] The server receives the personal data uploaded by the user, encrypts the data immediately after receiving it, and stores it securely in a database or cloud storage, ensuring the confidentiality and integrity of the data.

[1402] Data Preprocessing

[1403] The server performs preprocessing on the stored data. This preprocessing includes removing unnecessary spaces and symbols in the case of text data, and converting audio data into text using a speech recognition API. This converts the data into a unified format, allowing for smooth subsequent processing.

[1404] Feature extraction

[1405] After preprocessing is complete, the server uses natural language processing technology to extract features from the data. Specifically, it analyzes keywords, phrases, and grammatical structures in the text to identify the deceased's writing style and speaking patterns. In the case of audio data, it also extracts the rhythm and intonation of speech, as well as specific pronunciation patterns.

[1406] Model training

[1407] After the features are extracted, the server trains an AI model based on the extracted features. For example, a Transformer-based model is built and trained using a machine learning library. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge. This trained AI model is then used to interact with the user.

[1408] Emotion Recognition and Response Generation

[1409] The emotion engine recognizes the user's emotions in response to input from the user. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the AI ​​model and reflected when generating responses. This allows for the generation of appropriate responses according to the user's emotional state, resulting in more human-like interactions.

[1410] Providing an interface

[1411] The server provides users with an interface (e.g., a chatbot or voice response system) that integrates the trained AI model and emotion engine. Through this interface, users can enjoy conversations based on the recreated features and emotion recognition of the deceased person.

[1412] Specific examples

[1413] 1. User Data Provision:

[1414] The user opens the application, selects and uploads the diary and audio files of the deceased person.

[1415] 2. Receiving and storing data:

[1416] The server receives the deceased person's diary files and audio files, encrypts the data, and stores it in cloud storage.

[1417] 3. Data preprocessing:

[1418] The server cleans the diary text, removing unnecessary spaces and symbols, and converts the audio file into text using a speech recognition API.

[1419] 4. Feature extraction:

[1420] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[1421] 5. Train the model:

[1422] The server uses a machine learning library to train a Transformer model based on the features.

[1423] 6. Emotion Recognition and Response Generation:

[1424] In response to text or voice input from the user, the server uses an emotion engine to identify the user's emotion, and the AI ​​model generates the optimal response based on that emotion.

[1425] 7. Providing an interface:

[1426] The server provides a chatbot that integrates a trained model and an emotion engine, making it accessible to users.

[1427] 8. Communication:

[1428] The user types, "How are you feeling today?" and the server's AI model recognizes the user's emotion of joy and replies, "I feel great today," in the style of the deceased.

[1429] In this way, the system can not only accurately reproduce the characteristics and conversational style of the deceased, but also identify the user's emotions and provide appropriate responses based on those emotions, enabling richer communication.

[1430] The processing flow will be explained below.

[1431] Step 1:

[1432] The user selects and uploads personal data related to the deceased (e.g., text files, audio files) through a dedicated application or web interface.

[1433] Step 2:

[1434] The terminal transmits the data selected by the user to the server, and before transmitting the data, it is encrypted as necessary.

[1435] Step 3:

[1436] The server receives the data sent from the device, then immediately encrypts it again and stores it securely in a database or cloud storage.

[1437] Step 4:

[1438] The server starts preprocessing the stored data. For text data, unnecessary spaces and special characters are deleted and the format is standardized. For audio data, a speech recognition API is used to convert it into text data.

[1439] Step 5:

[1440] The server uses natural language processing technology to extract features from the preprocessed data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the characteristics of the deceased. In the case of audio data, it also extracts the rhythm and intonation of speech, as well as specific pronunciation patterns.

[1441] Step 6:

[1442] The server uses the extracted features to train an AI model using a machine learning library, such as building and training a Transformer-based model. During the training process, the server iterates over the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge.

[1443] Step 7:

[1444] The server uses an emotion engine to identify the emotion of user input (text and voice). The emotion engine performs text and voice analysis to determine emotional states such as joy, sadness, and anger.

[1445] Step 8:

[1446] The server adjusts the AI ​​model's response based on the emotional information obtained from the emotion engine. By incorporating the emotional information into response generation, an appropriate response is generated according to the user's emotional state.

[1447] Step 9:

[1448] The server provides an interface that integrates trained AI models and emotion engines. For example, users can interact with AI through chatbots or voice response systems.

[1449] Step 10:

[1450] When a user enters a question or message through the interface, it is sent to the server's AI model and emotion engine. The server uses the emotion engine to identify the user's emotion and uses the AI ​​model to generate an appropriate response. This response is then sent back to the user.

[1451] Step 11:

[1452] The device displays the response received from the server to the user. The user can also send questions or messages again, allowing for a continuous dialogue. This allows the user to experience a more human-like dialogue through an AI model that recognizes emotions and recreates the characteristics of the deceased.

[1453] Example 2

[1454] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1455] A system that enables conversation with the deceased must accurately reproduce the characteristics and conversational style of the deceased, recognize the user's emotions, and reflect them in the conversation. It must also convert various data formats into a unified format and efficiently extract features while ensuring the confidentiality and integrity of personal data.

[1456] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1457] In this invention, the server includes: a means for a user to upload personal data related to the deceased; a means for receiving and saving the uploaded personal data; a means for preprocessing the saved personal data, such as deleting unnecessary spaces and symbols in the case of text data and converting audio data into text using speech recognition technology; a means for extracting features from the preprocessed data; a means for training an AI model based on the extracted features to reproduce the deceased's unique conversational style; a means for recognizing emotions in response to user input and generating appropriate responses according to those emotions; and a means for providing an interface that allows the user to communicate with the deceased using the trained AI model and emotion engine. This makes it possible to reproduce the deceased's characteristics and conversational style while generating responses according to the user's emotions.

[1458] A "user" is a person who wishes to use the system to interact with the deceased.

[1459] A "deceased person" is someone who has already passed away, and the system will recreate that person's characteristics and conversation style.

[1460] "Personal data" is information related to a deceased person that is uploaded by a user, including text files, audio files, diaries, message history, etc.

[1461] "Means for uploading" refers to the method by which a user provides personal data relating to the deceased person to the system, including a dedicated application or web interface.

[1462] "Means of receiving and storing" refers to the process by which the server receives the uploaded personal data and stores it safely.

[1463] "Preprocessing" refers to processing performed on stored personal data, and includes deleting unnecessary spaces and symbols in the case of text data, and converting audio data into text using voice recognition technology.

[1464] "Features" are useful information extracted from personal data, and include keywords, phrases, grammatical structures, speech rhythm and intonation, etc.

[1465] An "AI model" is a mathematical model built and trained using machine learning techniques to recreate the speaking style and characteristics of a deceased person.

[1466] An "emotion engine" is a technology that recognizes emotions in response to user input and generates a response that corresponds to that emotion.

[1467] An "interface" is an interactive environment that allows users to communicate with the deceased using trained AI models and emotion engines, including chatbots and voice response systems.

[1468] A "uniform format" is a state in which data has been converted into a common format through preprocessing, allowing subsequent processing to proceed smoothly.

[1469] The present invention includes a system that reproduces the characteristics and patterns of the deceased and enables communication between the deceased and the user, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. This system is realized mainly through the process of providing personal data by the user, data processing by the server, learning of an AI model, and emotion recognition and response generation.

[1470] System configuration

[1471] 1. User Interface

[1472] Users upload personal data relating to the deceased using a dedicated application or web interface, which is designed to be easy for users to navigate and guide them through the selection and upload process.

[1473] 2. Receipt and storage of data

[1474] The server receives personal data uploaded by users, encrypts the data immediately after receiving it, and stores it securely in a database or cloud storage, thus ensuring the confidentiality and integrity of the received data.

[1475] 3. Data Preprocessing

[1476] The server performs preprocessing on the stored data. For text data, unnecessary spaces and symbols are removed, and for audio data, speech recognition technology (e.g., Google Speech-to-Text API) is used to convert it into text. This preprocessing converts the data into a unified format, allowing for smooth subsequent processing.

[1477] 4. Feature Extraction

[1478] The server extracts features from the pre-processed data using natural language processing technology (e.g., spaCy). It analyzes keywords, phrases, and grammatical structures in the text to identify the deceased's style and speaking patterns. In the case of audio data, speech rhythm, intonation, and specific pronunciation patterns are also extracted.

[1479] 5. Training the Model

[1480] The server trains an AI model based on the extracted features. For example, it builds and trains a Transformer-based model using a machine learning library (e.g., TensorFlow, PyTorch). During the training process, iterative processing is performed on the provided data, and the model acquires the ability to reproduce the deceased's unique speaking style and knowledge.

[1481] 6. Emotion Recognition and Response Generation

[1482] The emotion engine recognizes the user's emotions in response to input from the user. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the AI ​​model and reflected when generating responses. This generates appropriate responses according to the user's emotional state, resulting in more human-like interactions.

[1483] 7. Providing an Interface

[1484] The server provides an interface that integrates a trained AI model and an emotion engine. Through this interface (e.g., a chatbot or voice response system), users can enjoy conversations based on the recreated features and emotion recognition of the deceased.

[1485] Specific examples

[1486] 1. User Data Provision:

[1487] Users open the application, select and upload the deceased person's diary and audio files.

[1488] 2. Receiving and storing data:

[1489] The server receives the deceased person's diary files and audio files, encrypts the data, and stores it in cloud storage.

[1490] 3. Data preprocessing:

[1491] The server cleans the diary text, removing unnecessary spaces and symbols, and converts the audio file to text using the Google Speech-to-Text API.

[1492] 4. Feature extraction:

[1493] The server performs text analysis to extract keywords and unique expressions frequently used by the deceased.

[1494] 5. Train the model:

[1495] The server uses TensorFlow to train a Transformer model based on the features.

[1496] 6. Emotion Recognition and Response Generation:

[1497] In response to text or voice input from the user, the server uses an emotion engine to identify the user's emotion, and the AI ​​model generates the optimal response based on that emotion.

[1498] 7. Providing an interface:

[1499] The server provides a chatbot that integrates a trained model and an emotion engine, making it accessible to users.

[1500] 8. Communication:

[1501] The user types, "How are you feeling today?" and the server's AI model recognizes the user's emotion of joy and replies, "I feel great today," in the style of the deceased.

[1502] This system can not only reproduce the characteristics and conversational style of the deceased, but also identify the user's emotions and generate appropriate responses based on those emotions, enabling richer communication.

[1503] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1504] Step 1: Provide user data

[1505] The user opens a dedicated application or web interface and selects and uploads personal data related to the deceased (e.g., diary file "diary.txt" or audio file "voice_recording.mp3").

[1506] Input: User-selected text and audio files

[1507] Output: The uploaded file is sent to the server as input.

[1508] Step 2: Receiving and storing data

[1509] The server receives the personal data uploaded by the user, encrypts it with the AES-256 algorithm, and stores it in cloud storage.

[1510] Input: User-uploaded text and audio files

[1511] Output: The encrypted data is stored in cloud storage.

[1512] Step 3: Preprocessing the data

[1513] The server loads the stored data from cloud storage. For text data, it uses regular expressions to remove unnecessary spaces and symbols. For audio data, it uses the Google Speech-to-Text API to convert the audio to text.

[1514] Input: Encrypted text and audio data stored in cloud storage

[1515] Output: Clean text data and converted text data

[1516] Step 4: Feature extraction

[1517] The server extracts features from the preprocessed data using natural language processing technology (e.g., spaCy). Specifically, it analyzes keywords, phrases, grammatical structures, and speech rhythm and intonation in the text.

[1518] Input: Preprocessed text data

[1519] Output: Extracted keywords, phrases, grammatical structures, speech rhythm, intonation, and other features

[1520] Step 5: Training the model

[1521] The server trains an AI model based on the features. It uses a machine learning library (e.g., TensorFlow, PyTorch) to build and train a Transformer-based model.

[1522] Input: extracted features

[1523] Output: Trained AI model

[1524] Step 6: Emotion recognition and response generation

[1525] The user inputs a message into the system. The emotion engine on the server analyzes the user's input message and identifies the emotional state (e.g., joy, sadness, anger, etc.). The AI ​​model generates an appropriate response based on the emotion.

[1526] Input: Input message from the user

[1527] Output: Response message generated by the AI ​​model according to the emotion

[1528] Step 7: Providing an Interface

[1529] The server provides an interface (e.g., a chatbot) that integrates a trained AI model and an emotion engine. Users can access this interface through a browser or application.

[1530] Input: trained AI model, emotion engine, user message

[1531] Output: The interface presented to the user

[1532] Step 8: Communicate

[1533] The user initiates a dialogue with the deceased, and the server analyzes the user's input and generates a response based on their emotional state. For example, if the user types, "How are you feeling today?", the server's AI model will recognize the emotion of joy and generate the response, "I feel great today."

[1534] Input: User interaction message

[1535] Output: An appropriate response message based on the sentiment

[1536] This allows the characteristics and conversation style of the deceased to be reproduced, enabling more realistic communication that responds to the user's emotions.

[1537] (Application example 2)

[1538] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1539] In current virtual stores, it is difficult to provide personalized product recommendations to users, and there are currently no systems that provide an emotional shopping experience based on personal data related to the deceased. Furthermore, there is a lack of interfaces that can recognize the user's emotions and respond or provide recommendations accordingly. Therefore, there is a need for a system that can recognize the user's emotional state and generate optimal responses based on that, especially for a virtual store that can provide emotionally rich communication through dialogue that reflects the characteristics of the deceased.

[1540] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1541] In this invention, the server includes means for a user to upload individual data related to the deceased, means for receiving and storing the uploaded individual data, means for extracting feature data from the stored individual data, means for training a generative AI model based on the extracted feature data, means for recognizing the user's emotional state using an emotion engine, means for providing an interface that allows the user to communicate with the deceased in an emotionally adapted manner using the trained generative AI model and the emotion engine, and means for making product suggestions through dialogue in a virtual store that reflects the characteristics of the deceased. This allows the user to receive personalized product suggestions through emotion-based communication with the deceased.

[1542] "User" refers to an individual who uses the system to upload personal data related to the deceased and communicate in the virtual store.

[1543] A "deceased person" refers to an entity that had a past relationship with a user of the system, and is recreated by providing a basis for individual data related to the deceased.

[1544] "Individual data" is a general term for digital information that includes unique information related to the deceased, such as diaries, message history, and audio files.

[1545] "Uploading means" refers to the hardware and software that allows users to transfer individual data to the server.

[1546] "Means for receiving and storing" refers to the function for receiving uploaded individual data on the server side and storing it safely.

[1547] "Characteristic data" is data extracted from the stored individual data that indicates the unique characteristics of the deceased, such as their sentence structure and pronunciation patterns.

[1548] "Learning means" refers to a method and apparatus for training a generative AI model based on extracted feature data to reproduce the features of the deceased.

[1549] "Generative AI model" refers to an artificial intelligence model that is trained using machine learning algorithms to reproduce the conversational style and responses of a deceased person.

[1550] "Emotion Engine" refers to algorithms and software for recognizing a user's emotional state and generating responses based on that.

[1551] "Interface" refers to the user interaction environment that allows users to communicate with the deceased using the generative AI model and emotion engine.

[1552] A "virtual store" refers to a store-like environment that exists in a virtual space, and is a virtual platform where users can receive product suggestions and make purchases.

[1553] "Product suggestion means" refers to a method and device for reflecting the characteristics of the deceased person in a virtual store and suggesting appropriate products to the user.

[1554] This invention includes a system that reproduces the characteristics and patterns of the deceased and allows the user to communicate with the deceased, as well as an emotion engine that recognizes the user's emotions and reflects those emotions in the dialogue. The system's primary purpose is to suggest products through dialogue that reflects the characteristics of the deceased in a virtual store.

[1555] Generating a Program

[1556] This system is realized based on the following program configuration.

[1557] 1. User data provision:

[1558] Using a dedicated application, users upload personal data related to the deceased, such as diary entries, message history, or audio recordings.

[1559] 2. Receiving and storing data:

[1560] The server receives the individual data uploaded by users and securely stores it in a database or cloud storage. The data is encrypted immediately after receiving it to ensure confidentiality and integrity of the data.

[1561] 3. Data preprocessing:

[1562] The server performs preprocessing on the stored data, such as deleting unnecessary spaces and symbols in the case of text data, or converting audio data into text using a speech recognition API, thereby converting the data into a unified format.

[1563] 4. Extracting feature data:

[1564] After the preprocessing is complete, the server uses natural language processing technology to extract feature data. Specifically, it analyzes keywords, phrases, grammatical structures, etc. in the text to identify the deceased's writing style and speaking patterns.

[1565] 5. Training the generative AI model:

[1566] After the feature data is extracted, the server trains a generative AI model based on the extracted feature data. For example, it uses a machine learning library to build and train a Transformer-based model. During the training process, iterative processing is performed on the provided data to acquire the ability to reproduce the deceased person's unique speaking style and knowledge.

[1567] 6. Emotion Recognition and Response Generation:

[1568] The emotion engine recognizes the user's emotions in response to user input. The emotion engine identifies emotional states (e.g., joy, sadness, anger, etc.) from both text and voice data. The recognized emotions are fed back to the generative AI model and reflected when generating responses. This allows for the generation of appropriate responses according to the user's emotional state.

[1569] 7. Virtual store interface:

[1570] The server provides an interface that integrates a trained generative AI model and an emotion engine. Through smart glasses, users can enjoy conversations based on the deceased's recreated features and emotion recognition within a virtual store. For example, if a user types, "What do you think of this red dress?", the server's AI model will recognize the user's emotion and reply in the deceased's style, "This dress is vibrant and lovely. I think it suits you well."

[1571] Hardware and software used

[1572] Hardware: Server, smart glasses

[1573] Software: Machine learning libraries (e.g., Transformers), speech recognition APIs, databases and cloud storage, emotion engines

[1574] Specific examples

[1575] Scenario: A user wears smart glasses and enjoys shopping while walking through a virtual store.

[1576] Example dialogue:

[1577] User: "What do you think about this red dress?"

[1578] Virtual Guide: "This dress is vibrant and beautiful. I think it would look great on you."

[1579] Prompt Sentence Examples

[1580] The following text recognizes the user's emotional state when they suggest a product and generates a response based on that emotion, continuing the dialogue while reflecting the characteristics and style of the deceased.

[1581] User: What do you think about this red dress?

[1582] Emotional state: Joy

[1583] Response: This dress is vibrant and beautiful. I think it looks great on you.

[1584] In this way, the system not only reproduces the characteristics and conversational style of the deceased with high accuracy, but also identifies the user's emotions and returns appropriate responses based on those emotions, enabling richer communication.

[1585] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1586] Step 1:

[1587] The user uploads personal data relating to the deceased person using the application.

[1588] Specifically, the user selects data such as diary entries, message history, and audio files and uploads them on the application. At this time, the application generates an API request to transfer the data to the server and sends the data.

[1589] Input: Individual data such as diary entries, message history, and audio files

[1590] Output: Data transfer to the server in the form of an API request

[1591] Step 2:

[1592] The server receives the uploaded individual data, securely encrypts it, and stores it in a database or cloud storage.

[1593] Specifically, the server immediately encrypts the data it receives and stores it in a database or cloud storage, ensuring the confidentiality and integrity of the data.

[1594] Input: Individual data transferred by the user

[1595] Output: Encrypted database or cloud storage

[1596] Step 3:

[1597] The server performs preprocessing on the stored individual data.

[1598] Specifically, in the case of text data, unnecessary spaces and symbols are removed, and in the case of voice data, it is converted into text data using a voice recognition API. By converting the data into a unified format, subsequent processing becomes smoother.

[1599] Input: Data from an encrypted database or cloud storage

[1600] Output: Preprocessed data in a unified format

[1601] Step 4:

[1602] The server extracts feature data from the preprocessed data.

[1603] Specifically, natural language processing techniques are used to analyze keywords, phrases, and grammatical structures in text to identify the deceased's writing style and speaking patterns, and in the case of audio data, to extract speech rhythm, intonation, and specific pronunciation patterns.

[1604] Input: Data converted into a unified format

[1605] Output: characteristic data of the deceased

[1606] Step 5:

[1607] The server trains a generative AI model based on the extracted feature data.

[1608] Specifically, a Transformer-based AI model is built and trained using machine learning libraries (e.g., Transformers). During the training process, the model iterates on the provided data to gain the ability to reproduce the deceased person's unique speaking style and knowledge.

[1609] Input: Feature data

[1610] Output: A trained generative AI model

[1611] Step 6:

[1612] The user provides input through an interface to interact with the generative AI model.

[1613] Specifically, using smart glasses in a virtual store, users can input questions or comments by voice or text, and this data is sent to a server.

[1614] Input: User questions and comments

[1615] Output: Transfer of input data to the server

[1616] Step 7:

[1617] The server uses an emotion engine to recognize the user's emotional state and generates an appropriate response using a generative AI model.

[1618] Specifically, the emotion engine identifies the emotional state (e.g., joy, sadness, etc.) from the user's input data (voice or text) and feeds that emotion back to the generative AI model, which then generates the optimal response by taking into account the characteristics of the deceased and the user's emotions.

[1619] Input: User input data, emotion engine recognition results

[1620] Output: An appropriate response based on the user's sentiment

[1621] Step 8:

[1622] The server provides the generated response to the user in the virtual store.

[1623] Specifically, the generated response is communicated to the user through the smart glasses. For example, if a user asks, "What do you think of this red dress?", the server's AI model will generate a response such as, "This dress is vibrant and beautiful. I think it suits you well," and present it to the user through the smart glasses.

[1624] Input: The generated response

[1625] Output: Presentation of response through smart glasses

[1626] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1627] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1628] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1629] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1630] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1631] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1632] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1633] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1634] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1635] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1636] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1637] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1638] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1639] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1640] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1641] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1642] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1643] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1644] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1645] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1646] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1647] The following is further disclosed regarding the above embodiment.

[1648] (Claim 1)

[1649] A means for users to upload personal data relating to the deceased;

[1650] means for receiving and storing uploaded personal data;

[1651] A means for extracting features from the stored personal data;

[1652] A means of training an AI model based on the extracted features,

[1653] A system including a means for providing an interface that allows a user to communicate with the deceased using the trained AI model.

[1654] (Claim 2)

[1655] 2. The system of claim 1, further comprising means for pre-processing the stored personal data and converting it into a unified format.

[1656] (Claim 3)

[1657] 10. The system of claim 1, wherein the personal data includes text data and voice data, and further comprising means for converting voice data to text.

[1658]

[1659] "Example 1"

[1660] (Claim 1)

[1661] A means for users to upload personal data relating to the deceased;

[1662] means for receiving and storing uploaded personal data;

[1663] A means of encrypting and storing personal data in cloud storage or databases;

[1664] means for pre-processing the stored personal data and converting it into a unified format;

[1665] A means for extracting features from the preprocessed personal data;

[1666] A means of training an AI model based on the extracted features,

[1667] A means for providing an interface that allows a user to communicate with the deceased using the trained AI model; and

[1668] A system that includes a means for responding to user questions and messages through a provided interface.

[1669] (Claim 2)

[1670] The personal data, including text data and voice data, is further converted into text using voice recognition technology.

[1671] 10. The system of claim 1.

[1672] (Claim 3)

[1673] The method further includes a means for constructing and training a Transformer-based model using the extracted data features.

[1674] 10. The system of claim 1.

[1675] "Application Example 1"

[1676] (Claim 1)

[1677] A means for users to upload personal data relating to the deceased;

[1678] means for receiving and storing uploaded personal data;

[1679] A means for extracting features from the stored personal data;

[1680] A means for training a generative AI model based on the extracted features;

[1681] A means for providing an interface that allows a user to communicate with the deceased using the trained AI model; and

[1682] A means to recreate the voice and speaking style of the deceased based on the responses of the generative AI model, and

[1683] a means for inputting a prompt sentence for the user to enjoy a dialogue with the deceased;

[1684] means for performing preprocessing to convert voice data into text data;

[1685] A system including:

[1686] (Claim 2)

[1687] 10. The system of claim 1, further comprising means for pre-processing the stored personal data and converting it into a unified format.

[1688] (Claim 3)

[1689] 10. The system of claim 1, wherein the personal data includes text data and voice data, and further comprising means for converting voice data to text.

[1690] "Example 2: Combining Emotion Engines"

[1691] (Claim 1)

[1692] a means for a user to upload personal data relating to the deceased;

[1693] means for receiving and storing uploaded personal data;

[1694] A means for preprocessing the stored personal data, removing unnecessary spaces and symbols in the case of text data, and converting voice data into text using voice recognition technology;

[1695] means for extracting features from the preprocessed data;

[1696] A method for training an AI model based on the extracted features to reproduce the unique conversation style of the deceased, and

[1697] means for recognizing emotions in response to user input and generating an appropriate response in response to the emotions;

[1698] A system including means for providing an interface that allows a user to communicate with the deceased using the trained AI model and emotion engine.

[1699] (Claim 2)

[1700] 10. The system of claim 1, further comprising means for pre-processing the stored personal data and converting it into a unified format.

[1701] (Claim 3)

[1702] 10. The system of claim 1, wherein the personal data includes text data and voice data, and further comprising means for converting voice data to text.

[1703] "Application example 2 when combining emotion engines"

[1704] (Claim 1)

[1705] a means for users to upload personal data relating to the deceased;

[1706] means for receiving and storing the uploaded individual data;

[1707] means for extracting feature data from the stored individual data;

[1708] A means for training a generative AI model based on the extracted feature data;

[1709] a means for recognizing the emotional state of a user using an emotion engine;

[1710] a means for providing an interface that allows a user to engage in emotionally adapted communication with the deceased using the learned generative AI model and emotion engine; and

[1711] A system including a means for suggesting products through dialogue that reflects the characteristics of the deceased person in a virtual store.

[1712] (Claim 2)

[1713] 10. The system of claim 1, further comprising means for pre-processing the stored individual data and converting it into a standard format.

[1714] (Claim 3)

[1715] 2. The system of claim 1, wherein the individual data includes text data and voice data, and further comprising means for converting voice data to text. [Explanation of symbols]

[1716] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for users to upload personal data relating to the deceased; means for receiving and storing uploaded personal data; A means for extracting features from the stored personal data; A means of training an AI model based on the extracted features, A system including a means for providing an interface that allows a user to communicate with the deceased using the trained AI model.

2. The system of claim 1 , further comprising means for pre-processing the stored personal data and converting it into a unified format.

3. 2. The system of claim 1, wherein the personal data includes text data and voice data, and further comprising means for converting voice data to text.

Images