System

Abstract

A system is provided.SOLUTION: A system including means for uploading and storing voice data and text data before death, means for converting the voice data into text data and analyzing the data by using a natural language processing technology, means for training an AI model on the basis of an analysis result and learning characteristics of wording and speaking style, means for converting text generated by using the learned AI model into voice data, and means for providing an interface for performing interaction with a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention relates to a system that utilizes records of a deceased person while they were alive to provide a conversational experience that makes it seem as if the person is still alive. Conventional technologies have been inadequate in methods for recreating a conversational experience with a deceased person, making it difficult to faithfully reproduce the deceased's voice, speaking style, and phrasing. In particular, the quality of real-time responses and conversations has been low, making it impossible to provide a satisfying experience to users. It is necessary to solve these issues and provide a more natural and realistic conversational experience. [Means for solving the problem]

[0005] The present invention aims to solve the above problems by the following means.

[0006] By providing a system including a means for uploading and storing voice and text data from the deceased's lifetime, a means for converting the voice data into text data and analyzing the data using natural language processing technology, a means for training an AI model based on the analysis results and learning the characteristics of phrasing and speaking style, a means for converting the generated text into voice data using the trained AI model, and a means for providing an interface for dialogue with a user, it is possible to faithfully reproduce the voice and speaking style of the deceased and achieve a natural and realistic dialogue experience. Furthermore, by using a secure transmission protocol when uploading the voice and text data and including a means for generating responses and conducting dialogue in real time based on user input, it is possible to provide a safe and prompt dialogue experience.

[0007] "Audio data" refers to data in which an audio signal is recorded in digital format.

[0008] "Text data" is data recorded as character information.

[0009] "Uploading" is the act of a user sending data that they own to a server.

[0010] "Storage" means recording data in a database or file system for long-term retention.

[0011] "Conversion" is the process of changing data from one format to another.

[0012] "Natural language processing technology" is a technology that uses computers to understand, analyze, and generate human language.

[0013] "Analysis" means examining the contents of data in detail to clarify its structure and meaning.

[0014] An "AI model" is a mathematical model used to realize artificial intelligence.

[0015] "Training" is the process of training an AI model to learn large amounts of data and improve its performance.

[0016] A "phrase" is a particular combination of words or a way of expressing something.

[0017] A "user" is a person who uses the system.

[0018] An "interface" is the means or method by which a user interacts with a system.

[0019] "Real-time" means that data processing and response are immediate. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention is a system that uses records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased and realizes real-time conversation that reflects the characteristics of the deceased. Specific embodiments of the system are described below.

[0042] System Configuration

[0043] 1. Data Collection

[0044] The user uploads the deceased's audio data (e.g., call records) and text data (e.g., social media posts) to the system.

[0045] The terminal receives data provided by the user and transmits it to the server.

[0046] 2. Data Analysis

[0047] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[0048] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[0049] 3. Training the AI ​​model

[0050] The server trains the AI ​​model based on the analyzed results.

[0051] The AI ​​model is designed to learn and mimic the characteristic phrases and speaking styles of the deceased.

[0052] 4. Speech Synthesis

[0053] The server uses a trained AI model to convert the generated text into audio data.

[0054] This audio recreates the voice and speaking style of the deceased.

[0055] 5. User interaction interface

[0056] The terminal provides an interface for users to interact with each other, including a chat window and a voice input function.

[0057] The user initiates a dialogue with the deceased person through the interface.

[0058] Specific examples

[0059] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[0060] 1. The user types "Hello, Grandpa."

[0061] The terminal receives this text and sends it to the server.

[0062] 2. The server receives the text and generates a response using an AI model.

[0063] Based on training data from the deceased, the AI ​​model generates the response, "Hello, grandson."

[0064] 3. The server converts the generated text response into audio data.

[0065] The server uses a speech synthesis engine to create voice data saying "Hello, my grandson."

[0066] 4. The server sends the audio data to the device.

[0067] The terminal plays the received audio data and delivers it to the user.

[0068] This allows users to have an experience that feels as if they are having a conversation with the deceased. This system is designed to keep the memory of the deceased alive in the real world as a voice, and can provide users with a moving experience.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] A user uploads the deceased person's voice and text data into the system.

[0072] The user selects the data from a dedicated application or web interface and presses the upload button.

[0073] The terminal receives the uploaded data and transmits it to the server using a secure transmission protocol (e.g., SSL / TLS).

[0074] Step 2:

[0075] The server validates the received data and stores it in the database.

[0076] The server checks the data for consistency and integrity and ensures that it is in the correct format.

[0077] The server stores the verified data in a database.

[0078] Step 3:

[0079] The server runs the voice data through an automatic speech recognition (ASR) engine and converts it into text data.

[0080] The server passes the voice data to an ASR engine, which converts the voice signal into corresponding text.

[0081] The server temporarily stores the converted text data.

[0082] Step 4:

[0083] The server runs the text data through a natural language processing (NLP) engine for analysis.

[0084] The server passes the text data to an NLP engine for topic analysis, stylistic analysis, and extraction of features of the deceased's phrasing.

[0085] The server stores the analysis results and uses them to train the AI ​​model.

[0086] Step 5:

[0087] The server trains the AI ​​model.

[0088] The server uses the analysis results to input training data into the AI ​​model, allowing it to learn the characteristic speaking style and phrasing of the deceased.

[0089] The server stores the trained AI model.

[0090] Step 6:

[0091] The server receives user input and generates a response using an AI model.

[0092] The terminal transmits the text entered by the user to the server.

[0093] The server inputs the received text into an AI model to generate an appropriate response.

[0094] The server temporarily stores the generated text response.

[0095] Step 7:

[0096] The server generates a text response that is converted into audio data.

[0097] The server uses a text-to-speech (TTS) engine to convert the text response into audio data.

[0098] The server temporarily stores the generated voice data.

[0099] Step 8:

[0100] The server sends the audio data to the terminal.

[0101] The server sends the audio data to the terminal and delivers it to the user.

[0102] Step 9:

[0103] The terminal plays the audio data and delivers the response to the user.

[0104] The terminal uses an audio device to play the received audio data and plays it to the user.

[0105] These steps result in a system that recreates the voice and speaking style of the deceased, providing the user with a natural and immersive interactive experience.

[0106] Example 1

[0107] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0108] While photo albums, videos, and written documents have traditionally been used as a means of reminiscing about the deceased, these are one-way information sources and do not allow for two-way dialogue. Furthermore, there is no way to faithfully reproduce the voice and speaking style of the deceased, creating a need for technology that can provide a realistic conversational experience with the deceased. Furthermore, there has been a lack of technology that can provide a more realistic conversational experience by properly analyzing collected data and learning the specific phrases and speaking styles of the deceased.

[0109] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0110] In this invention, the server includes: means for a user to upload voice data and text data of the deceased to the system and transmit the received data to the server using a secure transmission protocol; means for the server to convert the voice data into text using an automatic speech recognition engine and use natural language processing technology to perform topic analysis, stylistic analysis, and extract phrasing features from the text data; means for training an AI model based on the analysis results and learning the phrasing and speaking characteristics; means for the server to convert the generated text using the trained AI model into voice data and reproduce the voice and speaking style of the deceased; and means for the terminal to provide an interface for dialogue with the user and for the user to start the dialogue, allowing the user to have a real-time dialogue with the deceased as if they were alive.

[0111] "Audio data" is digital data that records what the deceased person said.

[0112] "Text data" refers to written information stored in digital format, such as writings or social media posts by the deceased.

[0113] A "secure transmission protocol" is a communication protocol used to ensure security when sending and receiving data, and a specific example is SSL / TLS.

[0114] A "server" is a computer system that analyzes and processes data received from users.

[0115] An "automatic speech recognition engine" is a computer program that analyzes voice data and converts it into text data.

[0116] "Natural language processing technology" is a technology that analyzes text data and understands and extracts meaning and style.

[0117] "Topic analysis" is the process of extracting major topics or themes from text data.

[0118] "Style analysis" is the process of analyzing the writing style and phrasing of text data and extracting its characteristics.

[0119] An "AI model" is an algorithm that is trained using machine learning techniques to perform a specific task.

[0120] A "voice synthesis means" is a process or technology that takes text data as input and generates a specific voice.

[0121] An "interface" is a means or device for a user to interact with a system, and specifically includes a chat window and a voice input function.

[0122] This invention is a system that uses records of a deceased person to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes voice and text data to realize real-time conversations that reflect the characteristics of the deceased person.

[0123] System Configuration

[0124] Data collection

[0125] First, the user uploads the deceased's audio data (e.g., call records) or text data (e.g., social media posts) to the system. The user provides the data by dragging and dropping the audio or text file into a dedicated interface. The device then receives the data provided by the user and sends it to the server using a secure transmission protocol (e.g., SSL / TLS).

[0126] Data analysis

[0127] The received data is processed by the server. The server converts the audio data into text using an automatic speech recognition (ASR) engine (e.g., Google® Cloud Speech-to-Text). The converted text data is then analyzed using natural language processing (NLP) technology (e.g., TENSORFLOW® or spaCy). This analysis process involves topic analysis of the text data, stylistic analysis, and extraction of features of the deceased's phrasing.

[0128] Training an AI model

[0129] The server then trains an AI model based on the analyzed data. This AI model uses OpenAI's GPT-3 to learn the deceased's characteristic phrasing and speaking style. Training is performed on a server cluster equipped with high-performance GPUs, and training progress is monitored in real time.

[0130] Speech synthesis

[0131] Using the trained AI model, the server converts the generated text into speech data, which is generated using Amazon Polly, for example, and utilizes pre-trained voice characteristics of the deceased person to reproduce a more realistic voice.

[0132] User interaction interface

[0133] The device provides an interface for the user to interact with the deceased, including a chat window and voice input functionality. A form is provided for the user to enter text and a button is provided for recording voice commands. The user can then initiate a dialogue and experience the experience as if they were talking to the deceased person.

[0134] Specific examples

[0135] For example, if the user types "Hello, Grandpa" the following happens:

[0136] 1. The user types "Hello, Grandpa."

[0137] The device receives this text and sends it to the server using the HTTPS protocol.

[0138] 2. The server receives the text and generates a response using an AI model (e.g., GPT-3).

[0139] The AI ​​model generates the response, "Hello, my grandson." This process involves inputting a prompt to the model and selecting an appropriate response based on the output it generates.

[0140] 3. The server converts the generated text response into audio data.

[0141] The server uses Amazon Polly to create a voice message saying "Hello, my grandson." The generated voice message is then encoded into the appropriate format (e.g., MP3).

[0142] 4. The server sends the audio data to the device.

[0143] The device then plays the received audio data and delivers it to the user. The device uses its audio player function to play the audio, providing the user with an experience that makes it seem as if the deceased person is speaking.

[0144] This allows users to have a real-time conversation with the deceased as if they were alive, providing a moving experience. The system is designed to keep the memory of the deceased alive in the real world as their voice.

[0145] Prompt Sentence Examples

[0146] Below are some example prompts to input to a generative AI model:

[0147] User: Hello, Grandpa.

[0148] Deceased AI model: Hello, grandson. How are you today?

[0149] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0150] Step 1: Data collection

[0151] Users upload the audio and text data of the deceased to the system. Users provide data by dragging and dropping audio and text files into a dedicated interface (e.g., a web page or application).

[0152] Input: Audio and text data of the deceased

[0153] Output: Data uploaded to the device

[0154] The terminal receives the data provided by the user and sends it to the server using a secure transmission protocol (e.g. SSL / TLS).

[0155] Input: User-provided voice and text data

[0156] Output: Audio and text data sent to the server

[0157] Step 2: Data analysis

[0158] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text). The server receives the voice data as input and generates text data by performing speech recognition processing.

[0159] Input: Audio data

[0160] Output: Text data generated from the audio data

[0161] The server analyzes the generated text data using natural language processing (NLP) techniques (e.g., TensorFlow and spaCy). This analysis process involves topic and stylistic analysis of the text data to extract characteristics of the deceased's phrasing.

[0162] Input: Text data generated from audio data and text data uploaded by users

[0163] Output: Topic analysis results, stylistic analysis results, and phrasing features of text data

[0164] Step 3: Training the AI ​​model

[0165] The server then trains an AI model based on the analysis results. For example, it uses OpenAI's GPT-3 to learn the deceased's characteristic phrases and speaking style. The server receives the analyzed text data as input and uses it as training data for the AI ​​model.

[0166] Input: Topic analysis results, stylistic analysis results, and phrasing characteristics of text data

[0167] Output: A trained AI model

[0168] Step 4: Text-to-Speech

[0169] The server uses a trained AI model to convert the generated text into speech data, which is generated using Amazon Polly, for example, and uses pre-trained voice characteristics of the deceased person to create a realistic voice.

[0170] Input: Text generated by an AI model

[0171] Output: Generated audio data

[0172] Step 5: User interaction interface

[0173] The device provides an interface for user interaction, including a chat window and voice input functionality. It provides a form for the user to enter text and a button for recording voice commands.

[0174] Input: User request to start a conversation

[0175] Output: The interactive interface presented to the user

[0176] The user initiates a conversation with the deceased through the interface. For example, the user types "Hello, Grandpa" into a chat window, which the device then sends to the server. The server uses an AI model to generate a response, converts the generated text back into audio data, and sends it to the device. The device then plays the received audio data and delivers it to the user.

[0177] Input: Text entered by the user (e.g. "Hello, Grandpa")

[0178] Output: A response played in the deceased person's voice (e.g., "Hello, my grandson.")

[0179] (Application example 1)

[0180] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0181] While there are technologies that can revive the memories and interactive experiences of deceased people in the present day, there are no systems yet that allow users to enjoy these experiences in a virtual environment and receive recommendations for specific information and products. The present invention aims to improve the purchasing experience in a virtual environment by providing users with a personalized interactive experience by providing a conversational experience with a deceased person.

[0182] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0183] In this invention, the server includes a means for uploading and storing voice and text data from the deceased person's life, a means for converting the voice data into text data and analyzing the data using natural language processing technology, and a means for training an AI model based on the analysis results to learn the characteristics of the deceased person's phrasing and speaking style. This allows the user to recreate the deceased person's memories and conversations and receive information and product recommendations through an interactive experience in a virtual environment.

[0184] "Pre-mortem audio and text data" refers to audio and written records of the deceased person while they were alive.

[0185] "Means for uploading and storing" refers to the ability for users to import data into the system and store it on the server.

[0186] "Converting voice data into text data" refers to the process of converting voice data into text using voice recognition technology.

[0187] "Means of analyzing data using natural language processing technology" refers to a method of using NLP technology to analyze text data and extract information.

[0188] "Training an AI model" refers to the process of using machine learning algorithms to learn from analyzed data.

[0189] "Means of learning phrasing and speaking characteristics" refers to the ability of an AI model to mimic the speaking and expressive styles of a particular individual.

[0190] "Means for converting text into audio data" refers to speech synthesis technology that converts the generated text back into audio format.

[0191] "Means for providing dialogue to a user using a specific speech synthesis engine" refers to a method for responding to a user in real time by voice using a specified speech synthesis tool.

[0192] "Means for providing an interactive experience in a virtual environment" refers to means for allowing a user to interact using a virtual space.

[0193] "Means for recommending products and information" refers to a function that suggests specific products and information based on a dialogue with the user.

[0194] The present invention is a system that uses records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased and realizes real-time conversation that reflects the characteristics of the deceased. Specific embodiments of the system are described below.

[0195] System Configuration

[0196] 1. Data Collection:

[0197] The server provides a means for users to upload audio data (e.g., call logs) and text data (e.g., social media posts) of the deceased to the system, which is sent to the server using a secure transmission protocol.

[0198] The terminal receives data provided by the user and transmits it to the server.

[0199] 2. Data Analysis:

[0200] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text).

[0201] The server uses the text data and utilizes natural language processing (NLP) techniques (e.g., TensorFlow and PyTorch) to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[0202] 3. Training the AI ​​model:

[0203] The server uses the analyzed results to train an AI model that learns and mimics the characteristic phrases and speaking styles of the deceased.

[0204] 4. Speech synthesis:

[0205] The text generated by the trained AI model is converted into audio data using a speech synthesis engine (e.g., Amazon Polly), which recreates the voice and speaking style of the deceased.

[0206] 5. User interaction interface:

[0207] Provide an interface that allows users to interact with the deceased within a virtual environment (e.g., a virtual store built using Unity), including a chat window and voice input functionality.

[0208] Users are recommended products and information through simulated dialogue with the deceased.

[0209] Specific examples

[0210] For example, if a user asks "What are the features of this product?" in a virtual store, this text input goes through the following process:

[0211] 1. The user types, "What are the features of this product?"

[0212] The terminal receives this text and sends it to the server.

[0213] 2. The server receives the text and generates a response using an AI model.

[0214] Based on the training data of the deceased, the AI ​​model generates a response such as, "This product was created by founder XXX and is particularly rated for XXX. We believe it will be a perfect fit for your needs."

[0215] 3. The server converts the generated text response into audio data.

[0216] The server uses a speech synthesis engine to create voice data that says, "This product was created by our founder, XXX, and is particularly well-received for XXX. We believe it will perfectly meet your needs."

[0217] 4. The server sends the audio data to the terminal, which then plays the received audio data and delivers it to the user.

[0218] This allows users to have the experience of interacting with the deceased. This system allows the memory of the deceased to live on in the real world as a voice, providing users with a deeply moving experience.

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] A user uploads the deceased person's voice and text data into the system.

[0222] Input: Audio data (such as call records) and text data (such as social media posts) of the deceased.

[0223] Processing: The device receives these data and sends them to the server using a secure transmission protocol.

[0224] Output: Audio and text data received by the server.

[0225] Step 2:

[0226] The voice data received by the server is converted into text data using an automatic speech recognition (ASR) engine.

[0227] Input: Audio data uploaded to the server.

[0228] Processing: Convert the audio data into text using an ASR engine (e.g., Google Cloud Speech-to-Text).

[0229] Output: Text data.

[0230] Step 3:

[0231] The server uses the text data and analyzes the data using natural language processing (NLP) techniques.

[0232] Input: Text data converted from audio data.

[0233] Processing: The server uses NLP techniques (e.g., TensorFlow or PyTorch) to perform topic and stylistic analysis and extract features of the deceased's phrasing.

[0234] Output: Analysis results and characteristic phraseology data of the deceased.

[0235] Step 4:

[0236] The server trains the AI ​​model based on the analyzed results.

[0237] Input: Data obtained from NLP analysis.

[0238] Processing: The server trains the AI ​​model to learn the speech patterns and expressions of the deceased.

[0239] Output: The trained AI model.

[0240] Step 5:

[0241] The text generated using the trained AI model is converted into audio data.

[0242] Input: Input text from the user and the trained AI model.

[0243] Processing: The server uses the AI ​​model to generate response text, which is then converted into audio data using a speech synthesis engine (e.g., Amazon Polly).

[0244] Output: Audio data recreating the characteristics of the deceased.

[0245] Step 6:

[0246] It provides an interaction interface with the user within the virtual environment.

[0247] Input: User input text or voice.

[0248] Processing: The device responds to the user through a chat window or voice interaction interface within the virtual environment (e.g., a virtual store using Unity).

[0249] Output: The interactive experience provided to the user and the deceased's voice response.

[0250] Step 7:

[0251] Products and information are recommended through simulated dialogue between the user and the deceased.

[0252] Input: The user's question or request.

[0253] Processing: The server uses the AI ​​model to generate product recommendations and information based on the user's needs, and provides them in voice or text format.

[0254] Output: Providing recommended products and information, and improving the user's purchasing experience.

[0255] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0256] The present invention is a system that utilizes records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased to realize real-time conversations that reflect the characteristics of the deceased, and further combines it with an emotion engine that recognizes the user's emotions and adjusts responses, providing a more natural and emotionally rich conversational experience. Specific embodiments of the system are described below.

[0257] System Configuration

[0258] 1. Data Collection

[0259] The user uploads the deceased's audio data (e.g., call records) and text data (e.g., social media posts) to the system.

[0260] The terminal receives data provided by the user and transmits it to the server.

[0261] 2. Data Analysis

[0262] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[0263] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[0264] The server uses an emotion engine to recognize the user's emotions from the analyzed text data and voice data.

[0265] 3. Training the AI ​​model

[0266] The server trains the AI ​​model based on the analyzed results.

[0267] The AI ​​model is designed to learn and mimic the deceased person's characteristic phrasing, speaking style, and emotion recognition results.

[0268] 4. Speech Synthesis

[0269] The server uses a trained AI model to convert the generated text into audio data.

[0270] The audio reproduces the voice and speaking style of the deceased and is further customized to the user's emotional state.

[0271] 5. User interaction interface

[0272] The terminal provides an interface for users to interact with each other, including a chat window and a voice input function.

[0273] The user initiates a dialogue with the deceased person through the interface.

[0274] Specific examples

[0275] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[0276] 1. The user types "Hello, Grandpa."

[0277] The terminal receives this text and sends it to the server.

[0278] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[0279] The emotion engine determines whether the user is expressing emotions such as happiness, sadness, or surprise.

[0280] 3. The server generates a response using an AI model based on the user's emotions.

[0281] Based on the training data of the deceased and the emotion recognition results, the AI ​​model generates a response such as, "Hello, grandson. How are you feeling today?"

[0282] 4. The server converts the generated text response into audio data.

[0283] The server uses a speech synthesis engine to create audio data that says, "Hello, grandson. How are you feeling today?"

[0284] 5. The server sends the audio data to the device.

[0285] The terminal plays the received audio data and delivers it to the user.

[0286] This allows the user to have an experience that feels as if they are conversing with the deceased, and by providing responses that correspond to the user's emotions, it is possible to realize a more emotionally rich conversational experience. This system not only keeps the memory of the deceased alive in the real world as a voice, but also provides responses that take the user's emotions into consideration, thereby providing a moving experience for the user.

[0287] The processing flow will be explained below.

[0288] Step 1:

[0289] A user uploads the deceased person's voice and text data into the system.

[0290] The user selects the data from a dedicated application or web interface and presses the upload button.

[0291] The terminal receives the uploaded data and transmits it to the server using a secure transmission protocol (e.g., SSL / TLS).

[0292] Step 2:

[0293] The server validates the received data and stores it in the database.

[0294] The server checks the data for consistency and integrity and ensures that it is in the correct format.

[0295] The server stores the verified data in a database.

[0296] Step 3:

[0297] The server runs the voice data through an automatic speech recognition (ASR) engine and converts it into text data.

[0298] The server passes the voice data to an ASR engine, which converts the voice signal into corresponding text.

[0299] The server temporarily stores the converted text data.

[0300] Step 4:

[0301] The server runs the text data through a natural language processing (NLP) engine for analysis.

[0302] The server passes the text data to an NLP engine for topic analysis, stylistic analysis, and extraction of features of the deceased's phrasing.

[0303] The server stores the analysis results and uses them to train the AI ​​model.

[0304] Step 5:

[0305] The server runs the analyzed text and voice data through an emotion engine to recognize the user's emotions.

[0306] The server uses an emotion engine to determine the user's emotional state (e.g., happy, sad, angry, etc.) from the text and voice.

[0307] The server stores the emotion recognition results and uses them to generate responses.

[0308] Step 6:

[0309] The server trains the AI ​​model.

[0310] The server uses the analysis and emotion recognition results to input training data into the AI ​​model, allowing it to learn the deceased's characteristic speaking style, phrasing, and emotional state.

[0311] The server stores the trained AI model.

[0312] Step 7:

[0313] The server receives user input and generates a response using AI models and emotion engines.

[0314] The terminal transmits text and voice input from the user to the server.

[0315] The server analyzes the received text and voice to recognize the user's emotional state.

[0316] The server uses AI models to generate appropriate responses based on the characteristics of the deceased and the user's emotional state.

[0317] The server temporarily stores the generated text response.

[0318] Step 8:

[0319] The server generates a text response that is converted into audio data.

[0320] The server uses a text-to-speech (TTS) engine to convert the text response into audio data.

[0321] The server temporarily stores the generated voice data.

[0322] Step 9:

[0323] The server sends the audio data to the terminal.

[0324] The server sends the audio data to the terminal and delivers it to the user.

[0325] Step 10:

[0326] The terminal plays the audio data and delivers the response to the user.

[0327] The terminal uses an audio device to play the received audio data and plays it to the user.

[0328] These steps result in a system that recreates the voice and speaking style of the deceased, providing a natural and immersive conversational experience for the user. Furthermore, responses are generated based on the user's emotions, providing a more emotionally rich conversational experience.

[0329] Example 2

[0330] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0331] The present invention aims to solve the technical problem of reproducing the voice and speaking style of the deceased and providing responses that correspond to the user's emotions in a dialogue system that uses recordings of the deceased while they were alive. Conventional technologies have had problems in that it is difficult to provide real-time dialogue that fully reflects the characteristics of the deceased, and responses based on emotions are not provided, resulting in dialogue that is not natural and emotional.

[0332] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0333] In this invention, the server includes a means for identifying emotions from the analyzed data, a means for training an AI model based on the analysis results to learn characteristics of phrasing and speaking style, and a means for converting the generated text into voice data using the trained AI model, thereby making it possible to faithfully reproduce the characteristics of the deceased and generate responses according to the user's emotions in real time.

[0334] "Voice data from before death" refers to audio information recorded by the deceased person while they were alive.

[0335] "Text data" refers to data of sentences or words stored as character information.

[0336] "Upload" refers to the operation in which a user sends data from their own device to a server.

[0337] "Storage" means storing data in a fixed location so that it can be retrieved as needed.

[0338] "Automatic speech recognition (ASR)" is a technology that converts voice data into text data.

[0339] "Natural Language Processing (NLP)" is a technology that processes and analyzes human language using a computer.

[0340] An "emotion engine" is an algorithm or system for identifying a user's emotion from text or voice data.

[0341] An "AI model" is a mathematical model that uses machine learning algorithms to make predictions and classifications from data.

[0342] "Training" is the process by which an AI model learns patterns and features from data.

[0343] "Speech synthesis" is a technology that converts text data into voice data and reproduces the human voice.

[0344] An "interface" is a window or means through which a user and a system can interact with each other.

[0345] A "secure transmission protocol" is a set of rules and methods for securely communicating data.

[0346] "Real-time" refers to immediate response to user input.

[0347] MODE FOR CARRYING OUT THE INVENTION

[0348] This invention is a system that uses records of a deceased person to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased to realize real-time conversations that reflect the characteristics of the deceased, and by combining it with an emotion engine that recognizes the user's emotions and adjusts responses, it provides a more natural and emotionally rich conversational experience.

[0349] System Configuration

[0350] 1. Data Collection

[0351] The user uploads the deceased's voice data (e.g., call records) and text data (e.g., social media posts) to the system. The user selects the data using a smartphone or PC and presses the upload button through the system's interface.

[0352] The device receives the data provided by the user and transmits it to the server. The device transmits the uploaded data to the server over a secure connection.

[0353] 2. Data Analysis

[0354] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text), sends the audio file to a Google Cloud API, and retrieves the returned text data.

[0355] The server uses natural language processing (NLP) techniques (e.g., spaCy, NLTK) to analyze the text data, perform topic analysis, stylistic analysis, and extract features of the deceased's phrasing. For example, it analyzes the text of the deceased's social media posts to extract topics such as "travel," "family," and "work."

[0356] The server uses an emotion engine (e.g., IBM Watson® Tone Analyzer) to recognize the user's emotion from the analyzed text and voice data. For example, it recognizes the text "I'm very happy today" and extracts the emotion of joy.

[0357] 3. Training the AI ​​model

[0358] The server trains an AI model (e.g., GPT-3) based on the analyzed results. The server trains the AI ​​model using the deceased's phrasing and emotion recognition results to learn how the deceased spoke.

[0359] 4. Speech Synthesis

[0360] The server uses a trained AI model to convert the generated text into speech data (e.g., Amazon Polly), and then sends the generated response text to Amazon Polly to obtain an audio file.

[0361] The server sends the audio data to the terminal, which then sends the created audio data to the end user's terminal, where it is played.

[0362] 5. User interaction interface

[0363] The device provides an interface for the user to interact with the deceased, such as a chat window or voice input function (e.g., a web app or mobile app). The user initiates a conversation with the deceased through the interface.

[0364] The user can enter text into the chat window or speak by pressing the voice input button. For example, the user can enter "Hello, Grandpa," and the device will send this to the server.

[0365] Specific examples

[0366] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[0367] 1. The user types "Hello, Grandpa."

[0368] The terminal receives this text and sends it to the server.

[0369] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[0370] The emotion engine determines whether the user is expressing emotions such as happiness, sadness, or surprise.

[0371] 3. The server generates a response using an AI model based on the user's emotions.

[0372] Based on the training data of the deceased and the emotion recognition results, the AI ​​model generates a response such as, "Hello, grandson. How are you feeling today?"

[0373] 4. The server converts the generated text response into audio data.

[0374] The server uses a speech synthesis engine to create audio data that says, "Hello, grandson. How are you feeling today?"

[0375] 5. The server sends the audio data to the terminal, which then plays the received audio data and delivers it to the user.

[0376] This allows the user to have an experience that feels as if they are having a conversation with the deceased. Furthermore, responses are provided that correspond to the user's emotions, making for a more emotionally rich conversation experience.

[0377] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0378] Step 1:

[0379] Users upload audio and text data of the deceased.

[0380] Input: The user selects an audio file (e.g., call recording) or text file (e.g., social media post) of the deceased person from their smartphone or PC and uploads it to the system.

[0381] Specific operation: When the user clicks the upload button, voice and text data is sent to the system.

[0382] Output: Uploaded data is sent from the user's device to the system's device.

[0383] Step 2:

[0384] The terminal receives data provided by the user and transmits it to the server.

[0385] Input: Voice and text data sent from the user's device.

[0386] Specific operation: The device temporarily stores the received data and then transmits it to the server using a secure connection (e.g., HTTPS protocol).

[0387] Output: The audio and text data is sent to the server.

[0388] Step 3:

[0389] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[0390] Input: The audio data sent to the server.

[0391] What happens: The server calls the Google Cloud Speech-to-Text API to convert the audio file into text.

[0392] Output: The audio data is converted to text data.

[0393] Step 4:

[0394] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[0395] Input: Text data converted by the ASR engine.

[0396] How it works: The server uses spaCy and NLTK to analyze text data and identify specific themes, keywords, and writing styles. For example, analyzing the text of a deceased person's social media posts to extract topics such as travel and family.

[0397] Output: Metadata about the topic, stylistic, and phrasing features of the text.

[0398] Step 5:

[0399] The server uses an emotion engine to recognize the user's emotions from the analyzed text data and voice data.

[0400] Input: Text data after NLP processing.

[0401] How it works: The server uses IBM Watson Tone Analyzer to analyze the emotions contained in the text. For example, it recognizes the text "I'm so happy today" and extracts the emotion of joy.

[0402] Output: Emotion recognition results based on text data.

[0403] Step 6:

[0404] The server trains the AI ​​model based on the analyzed results.

[0405] Input: Emotion recognition results and data on the deceased person's characteristic phrasing and speaking style.

[0406] What it does: The server inputs this data into an AI model (e.g., GPT-3) for training.

[0407] Output: The trained AI model.

[0408] Step 7:

[0409] The server uses a trained AI model to convert the generated text into audio data.

[0410] Input: The text response generated by the trained AI model.

[0411] What happens: The server uses the Amazon Polly API to convert text to speech, for example, "Hello, grandson. How are you feeling today?" into an audio file.

[0412] Output: Audio data reproducing the voice of the deceased.

[0413] Step 8:

[0414] The server sends the audio data to the terminal.

[0415] Input: Speech data generated by a speech synthesis engine.

[0416] What it does: The server generates audio data and sends it to the device using a secure connection (e.g. SSL / TLS).

[0417] Output: The received audio data arrives at the device.

[0418] Step 9:

[0419] The terminal provides an interface with which the user can interact.

[0420] Input: Voice and text data sent to the device.

[0421] Specific operation: The device displays a chat window and voice input function, creating an environment for the user to interact.

[0422] Output: An interface with which the user can initiate an interaction.

[0423] Step 10:

[0424] The user initiates a dialogue with the deceased person through the interface.

[0425] Input: Text or speech entered by the user through the interface.

[0426] Specific operation: When the user starts a conversation by entering something like "Hello, Grandpa," the voice and text are sent via the device to the server, where they are processed again.

[0427] Output: Real-time dialogue with the deceased becomes possible.

[0428] (Application example 2)

[0429] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0430] The present invention relates to a system that provides a dialogue experience using the voice and text data of a deceased person. However, with conventional technologies, it has been difficult to faithfully reproduce the voice and phrasing of the deceased, and it has also been difficult to provide an emotionally rich dialogue experience with the user. Furthermore, there are no systems that can provide natural and emotional dialogue in virtual environments, particularly for specific applications such as product explanations and guidance. The present invention aims to solve these problems.

[0431] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0432] In this invention, the server includes means for uploading and storing voice data and text data from the deceased's life, means for converting the voice data into text data and analyzing the data using natural language processing technology, means for training an AI model based on the analysis results and learning the characteristics of phrasing and speaking style, means for converting the generated text into voice data using the trained AI model, means for providing an interface for dialogue with the user, means for explaining and guiding products in a virtual environment, and means for recognizing the user's emotional state and generating a response accordingly. This allows the user to have an experience that feels as if they are conversing with the deceased, and further enables effective product explanations and guidance in a virtual environment.

[0433] "Voice data and text data from the deceased's lifetime" refers to recordings of voices made by the deceased during their lifetime, as well as sentences and messages left in text format.

[0434] "Uploading and storing" refers to a user sending data to a server via a terminal and storing the data for a long period of time.

[0435] "Converting voice data into text data" refers to converting voice information into corresponding text information using voice recognition technology.

[0436] "Natural language processing technology" refers to technology that enables computers to understand and analyze language spoken naturally by humans.

[0437] "Analyzing data" refers to analyzing collected data and extracting meaning and patterns from it.

[0438] "Training an AI model" refers to having an artificial intelligence learn from collected data and improve its capabilities.

[0439] "Phraseology and speaking characteristics" refers to the specific language and expressions used by a particular person.

[0440] "Converting the generated text into voice data" means converting text data into voice using voice synthesis technology.

[0441] An "interface for interacting with a user" is a mechanism that provides input and output means for a user to interact with a system.

[0442] "Explaining and guiding products in a virtual environment" means providing information about products to users in a virtual space.

[0443] "Recognizing the user's emotional state and generating a response accordingly" means analyzing the user's emotions and generating an appropriate response based on the results.

[0444] This invention is a system that uses the voice and text data of the deceased to provide a conversational experience that makes it seem as if the deceased is still alive. This system realizes real-time conversation that reflects the deceased's distinctive voice and speaking style, as well as the user's emotions.

[0445] First, the user uploads the deceased's voice data (such as call records) and text data (such as social media posts) to the system using their device. The uploaded data is then securely stored on the server using a secure transmission protocol.

[0446] The server then converts the received audio data into text using an automatic speech recognition (ASR) engine (e.g., Google Speech-to-Text), which is then analyzed using natural language processing (NLP) techniques (e.g., spaCy) to extract features such as topic, style, and phrasing.

[0447] The server then uses the analysis results to train an AI model (e.g., OpenAI GPT-4®) that can mimic the deceased person's characteristic phrasing and speaking style and generate responses that correspond to the user's emotional state.

[0448] The server then uses the trained AI model to convert the generated text into audio data using a speech synthesis engine (e.g., Amazon Polly), which reproduces the voice and speaking style of the deceased and is further customized according to the user's emotions.

[0449] Finally, the device provides a dialogue interface for the user. Specifically, it includes a chat window and voice input functionality, allowing the user to initiate a dialogue with the deceased through the interface. Furthermore, the system is also equipped with the ability to explain and guide products within the virtual environment. When the user asks something, it can analyze their emotions and provide an appropriate response.

[0450] Specific examples

[0451] For example, if a user types "Please describe this product," this text input goes through the following process:

[0452] 1. The user types, "Please explain this product."

[0453] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[0454] 3. The server generates a response using an AI model based on the user's emotions.

[0455] 4. The server converts the generated text response into audio data.

[0456] 5. The server sends the audio data to the device, and the device plays the audio.

[0457] Example prompt sentence:

[0458] User: "Please explain this product"

[0459] To the AI ​​model:

[0460] User sentiment: 'Curiosity'

[0461] Using training data from a deceased person: "This is a product I love. It has the following features..."

[0462] Response generation and speech synthesis: "Amazon Polly"

[0463] This system not only allows users to have the experience of interacting with the deceased, but also enables effective product explanations and guidance within the virtual environment.

[0464] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0465] Step 1:

[0466] The device provides a means for users to upload audio and text data of the deceased. Users upload data, such as call logs and social media posts, to the device, which then securely transmits it to the server. The input data in this process is audio and text data, and the output data is the uploaded data stored on the server.

[0467] Step 2:

[0468] The server converts the uploaded voice data into text data using an automatic speech recognition (ASR) engine (e.g., Google Speech-to-Text). The input is voice data, which is converted into text through the ASR engine. The output is the converted text data.

[0469] Step 3:

[0470] The server analyzes the converted text data using natural language processing (NLP) techniques (e.g., spaCy). This analysis extracts features of topic, style, and phrasing. The input is the text data converted from speech, and the output is the analyzed text data and its features.

[0471] Step 4:

[0472] The server trains an AI model (e.g., OpenAI GPT-4) based on the analysis results. This training creates a model that reproduces the characteristic speech patterns and expressions of the deceased. The input is the analysis results, and the output is the trained AI model.

[0473] Step 5:

[0474] The server generates text responses using a trained AI model during user interaction. It also uses an emotion analysis engine (e.g., IBM Watson Emotion Analysis) to recognize the user's emotional state and tailor the response accordingly. The input is the user's text input and the result of the emotional state analysis, and the output is the generated response text.

[0475] Step 6:

[0476] The server uses a speech synthesis engine (e.g., Amazon Polly) to convert the generated text into the deceased person's voice. The input is the generated text response, and the output is the converted voice data.

[0477] Step 7:

[0478] The server sends the converted voice data to the terminal, and the terminal plays the voice data. The user listens to the played voice through the dialogue interface and makes the following input: The input is the voice data sent from the server, and the output is the voice played to the user.

[0479] Step 8:

[0480] The device provides a dialogue interface that explains and guides users through products in a virtual environment. When a user inputs a question about a product, the server uses emotion analysis and an AI model to generate an appropriate response and transmits it to the device as voice data. The input is the user's question and emotional state, and the output is voice data with the appropriate response.

[0481] Through this detailed process, users can engage in emotionally rich conversations with the deceased and experience product descriptions in a natural way in a virtual store.

[0482] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0483] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0484] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0485] [Second embodiment]

[0486] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0487] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0488] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0489] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0490] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0491] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0492] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0493] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0494] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0495] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0496] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0497] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0498] The present invention is a system that uses records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased and realizes real-time conversation that reflects the characteristics of the deceased. Specific embodiments of the system are described below.

[0499] System Configuration

[0500] 1. Data Collection

[0501] The user uploads the deceased's audio data (e.g., call records) and text data (e.g., social media posts) to the system.

[0502] The terminal receives data provided by the user and transmits it to the server.

[0503] 2. Data Analysis

[0504] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[0505] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[0506] 3. Training the AI ​​model

[0507] The server trains the AI ​​model based on the analyzed results.

[0508] The AI ​​model is designed to learn and mimic the characteristic phrases and speaking styles of the deceased.

[0509] 4. Speech Synthesis

[0510] The server uses a trained AI model to convert the generated text into audio data.

[0511] This audio recreates the voice and speaking style of the deceased.

[0512] 5. User interaction interface

[0513] The terminal provides an interface for users to interact with each other, including a chat window and a voice input function.

[0514] The user initiates a dialogue with the deceased person through the interface.

[0515] Specific examples

[0516] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[0517] 1. The user types "Hello, Grandpa."

[0518] The terminal receives this text and sends it to the server.

[0519] 2. The server receives the text and generates a response using an AI model.

[0520] Based on training data from the deceased, the AI ​​model generates the response, "Hello, grandson."

[0521] 3. The server converts the generated text response into audio data.

[0522] The server uses a speech synthesis engine to create voice data saying "Hello, my grandson."

[0523] 4. The server sends the audio data to the device.

[0524] The terminal plays the received audio data and delivers it to the user.

[0525] This allows users to have an experience that feels as if they are having a conversation with the deceased. This system is designed to keep the memory of the deceased alive in the real world as a voice, and can provide users with a moving experience.

[0526] The processing flow will be explained below.

[0527] Step 1:

[0528] A user uploads the deceased person's voice and text data into the system.

[0529] The user selects the data from a dedicated application or web interface and presses the upload button.

[0530] The terminal receives the uploaded data and transmits it to the server using a secure transmission protocol (e.g., SSL / TLS).

[0531] Step 2:

[0532] The server validates the received data and stores it in the database.

[0533] The server checks the data for consistency and integrity and ensures that it is in the correct format.

[0534] The server stores the verified data in a database.

[0535] Step 3:

[0536] The server runs the voice data through an automatic speech recognition (ASR) engine and converts it into text data.

[0537] The server passes the voice data to an ASR engine, which converts the voice signal into corresponding text.

[0538] The server temporarily stores the converted text data.

[0539] Step 4:

[0540] The server runs the text data through a natural language processing (NLP) engine for analysis.

[0541] The server passes the text data to an NLP engine for topic analysis, stylistic analysis, and extraction of features of the deceased's phrasing.

[0542] The server stores the analysis results and uses them to train the AI ​​model.

[0543] Step 5:

[0544] The server trains the AI ​​model.

[0545] The server uses the analysis results to input training data into the AI ​​model, allowing it to learn the characteristic speaking style and phrasing of the deceased.

[0546] The server stores the trained AI model.

[0547] Step 6:

[0548] The server receives user input and generates a response using an AI model.

[0549] The terminal transmits the text entered by the user to the server.

[0550] The server inputs the received text into an AI model to generate an appropriate response.

[0551] The server temporarily stores the generated text response.

[0552] Step 7:

[0553] The server generates a text response that is converted into audio data.

[0554] The server uses a text-to-speech (TTS) engine to convert the text response into audio data.

[0555] The server temporarily stores the generated voice data.

[0556] Step 8:

[0557] The server sends the audio data to the terminal.

[0558] The server sends the audio data to the terminal and delivers it to the user.

[0559] Step 9:

[0560] The terminal plays the audio data and delivers the response to the user.

[0561] The terminal uses an audio device to play the received audio data and plays it to the user.

[0562] These steps result in a system that recreates the voice and speaking style of the deceased, providing the user with a natural and immersive interactive experience.

[0563] Example 1

[0564] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0565] While photo albums, videos, and written documents have traditionally been used as a means of reminiscing about the deceased, these are one-way information sources and do not allow for two-way dialogue. Furthermore, there is no way to faithfully reproduce the voice and speaking style of the deceased, creating a need for technology that can provide a realistic conversational experience with the deceased. Furthermore, there has been a lack of technology that can provide a more realistic conversational experience by properly analyzing collected data and learning the specific phrases and speaking styles of the deceased.

[0566] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0567] In this invention, the server includes: means for a user to upload voice data and text data of the deceased to the system and transmit the received data to the server using a secure transmission protocol; means for the server to convert the voice data into text using an automatic speech recognition engine and use natural language processing technology to perform topic analysis, stylistic analysis, and extract phrasing features from the text data; means for training an AI model based on the analysis results and learning the phrasing and speaking characteristics; means for the server to convert the generated text using the trained AI model into voice data and reproduce the voice and speaking style of the deceased; and means for the terminal to provide an interface for dialogue with the user and for the user to start the dialogue, allowing the user to have a real-time dialogue with the deceased as if they were alive.

[0568] "Audio data" is digital data that records what the deceased person said.

[0569] "Text data" refers to written information stored in digital format, such as writings or social media posts by the deceased.

[0570] A "secure transmission protocol" is a communication protocol used to ensure security when sending and receiving data, and a specific example is SSL / TLS.

[0571] A "server" is a computer system that analyzes and processes data received from users.

[0572] An "automatic speech recognition engine" is a computer program that analyzes voice data and converts it into text data.

[0573] "Natural language processing technology" is a technology that analyzes text data and understands and extracts meaning and style.

[0574] "Topic analysis" is the process of extracting major topics or themes from text data.

[0575] "Style analysis" is the process of analyzing the writing style and phrasing of text data and extracting its characteristics.

[0576] An "AI model" is an algorithm that is trained using machine learning techniques to perform a specific task.

[0577] A "voice synthesis means" is a process or technology that takes text data as input and generates a specific voice.

[0578] An "interface" is a means or device for a user to interact with a system, and specifically includes a chat window and a voice input function.

[0579] This invention is a system that uses records of a deceased person to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes voice and text data to realize real-time conversations that reflect the characteristics of the deceased person.

[0580] System Configuration

[0581] Data collection

[0582] First, the user uploads the deceased's audio data (e.g., call records) or text data (e.g., social media posts) to the system. The user provides the data by dragging and dropping the audio or text file into a dedicated interface. The device then receives the data provided by the user and sends it to the server using a secure transmission protocol (e.g., SSL / TLS).

[0583] Data analysis

[0584] The received data is processed by the server. The server converts the audio data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text). The converted text data is then analyzed using natural language processing (NLP) techniques (e.g., TensorFlow or spaCy). This analysis process involves topic analysis of the text data, stylistic analysis, and extraction of features of the deceased's phrasing.

[0585] Training an AI model

[0586] The server then uses the analyzed data to train an AI model, such as OpenAI's GPT-3, which learns the deceased's characteristic phrasing and speaking style. Training is carried out on a server cluster equipped with high-performance GPUs, and training progress is monitored in real time.

[0587] Speech synthesis

[0588] Using the trained AI model, the server converts the generated text into speech data, which is generated using Amazon Polly, for example, and utilizes pre-trained voice characteristics of the deceased person to reproduce a more realistic voice.

[0589] User interaction interface

[0590] The device provides an interface for the user to interact with the deceased, including a chat window and voice input functionality. A form is provided for the user to enter text and a button is provided for recording voice commands. The user can then initiate a dialogue and experience the experience as if they were talking to the deceased person.

[0591] Specific examples

[0592] For example, if the user types "Hello, Grandpa" the following happens:

[0593] 1. The user types "Hello, Grandpa."

[0594] The device receives this text and sends it to the server using the HTTPS protocol.

[0595] 2. The server receives the text and generates a response using an AI model (e.g., GPT-3).

[0596] The AI ​​model generates the response, "Hello, my grandson." This process involves inputting a prompt to the model and selecting an appropriate response based on the output it generates.

[0597] 3. The server converts the generated text response into audio data.

[0598] The server uses Amazon Polly to create a voice message saying "Hello, my grandson." The generated voice message is then encoded into the appropriate format (e.g., MP3).

[0599] 4. The server sends the audio data to the device.

[0600] The device then plays the received audio data and delivers it to the user. The device uses its audio player function to play the audio, providing the user with an experience that makes it seem as if the deceased person is speaking.

[0601] This allows users to have a real-time conversation with the deceased as if they were alive, providing a moving experience. The system is designed to keep the memory of the deceased alive in the real world as their voice.

[0602] Prompt Sentence Examples

[0603] Below are some example prompts to input to a generative AI model:

[0604] User: Hello, Grandpa.

[0605] Deceased AI model: Hello, grandson. How are you today?

[0606] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0607] Step 1: Data collection

[0608] Users upload the audio and text data of the deceased to the system. Users provide data by dragging and dropping audio and text files into a dedicated interface (e.g., a web page or application).

[0609] Input: Audio and text data of the deceased

[0610] Output: Data uploaded to the device

[0611] The terminal receives the data provided by the user and sends it to the server using a secure transmission protocol (e.g. SSL / TLS).

[0612] Input: User-provided voice and text data

[0613] Output: Audio and text data sent to the server

[0614] Step 2: Data analysis

[0615] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text). The server receives the voice data as input and generates text data by performing speech recognition processing.

[0616] Input: Audio data

[0617] Output: Text data generated from the audio data

[0618] The server analyzes the generated text data using natural language processing (NLP) techniques (e.g., TensorFlow and spaCy). This analysis process involves topic and stylistic analysis of the text data to extract characteristics of the deceased's phrasing.

[0619] Input: Text data generated from audio data and text data uploaded by users

[0620] Output: Topic analysis results, stylistic analysis results, and phrasing features of text data

[0621] Step 3: Training the AI ​​model

[0622] The server then trains an AI model based on the analysis results. For example, it uses OpenAI's GPT-3 to learn the deceased's characteristic phrases and speaking style. The server receives the analyzed text data as input and uses it as training data for the AI ​​model.

[0623] Input: Topic analysis results, stylistic analysis results, and phrasing characteristics of text data

[0624] Output: A trained AI model

[0625] Step 4: Text-to-Speech

[0626] The server uses a trained AI model to convert the generated text into speech data, which is generated using Amazon Polly, for example, and uses pre-trained voice characteristics of the deceased person to create a realistic voice.

[0627] Input: Text generated by an AI model

[0628] Output: Generated audio data

[0629] Step 5: User interaction interface

[0630] The device provides an interface for user interaction, including a chat window and voice input functionality. It provides a form for the user to enter text and a button for recording voice commands.

[0631] Input: User request to start a conversation

[0632] Output: The interactive interface presented to the user

[0633] The user initiates a conversation with the deceased through the interface. For example, the user types "Hello, Grandpa" into a chat window, which the device then sends to the server. The server uses an AI model to generate a response, converts the generated text back into audio data, and sends it to the device. The device then plays the received audio data and delivers it to the user.

[0634] Input: Text entered by the user (e.g. "Hello, Grandpa")

[0635] Output: A response played in the deceased person's voice (e.g., "Hello, my grandson.")

[0636] (Application example 1)

[0637] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0638] While there are technologies that can revive the memories and interactive experiences of deceased people in the present day, there are no systems yet that allow users to enjoy these experiences in a virtual environment and receive recommendations for specific information and products. The present invention aims to improve the purchasing experience in a virtual environment by providing users with a personalized interactive experience by providing a conversational experience with a deceased person.

[0639] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0640] In this invention, the server includes a means for uploading and storing voice and text data from the deceased person's life, a means for converting the voice data into text data and analyzing the data using natural language processing technology, and a means for training an AI model based on the analysis results to learn the characteristics of the deceased person's phrasing and speaking style. This allows the user to recreate the deceased person's memories and conversations and receive information and product recommendations through an interactive experience in a virtual environment.

[0641] "Pre-mortem audio and text data" refers to audio and written records of the deceased person while they were alive.

[0642] "Means for uploading and storing" refers to the ability for users to import data into the system and store it on the server.

[0643] "Converting voice data into text data" refers to the process of converting voice data into text using voice recognition technology.

[0644] "Means of analyzing data using natural language processing technology" refers to a method of using NLP technology to analyze text data and extract information.

[0645] "Training an AI model" refers to the process of using machine learning algorithms to learn from analyzed data.

[0646] "Means of learning phrasing and speaking characteristics" refers to the ability of an AI model to mimic the speaking and expressive styles of a particular individual.

[0647] "Means for converting text into audio data" refers to speech synthesis technology that converts the generated text back into audio format.

[0648] "Means for providing dialogue to a user using a specific speech synthesis engine" refers to a method for responding to a user in real time by voice using a specified speech synthesis tool.

[0649] "Means for providing an interactive experience in a virtual environment" refers to means for allowing a user to interact using a virtual space.

[0650] "Means for recommending products and information" refers to a function that suggests specific products and information based on a dialogue with the user.

[0651] The present invention is a system that uses records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased and realizes real-time conversation that reflects the characteristics of the deceased. Specific embodiments of the system are described below.

[0652] System Configuration

[0653] 1. Data Collection:

[0654] The server provides a means for users to upload audio data (e.g., call logs) and text data (e.g., social media posts) of the deceased to the system, which is sent to the server using a secure transmission protocol.

[0655] The terminal receives data provided by the user and transmits it to the server.

[0656] 2. Data Analysis:

[0657] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text).

[0658] The server uses the text data and utilizes natural language processing (NLP) techniques (e.g., TensorFlow and PyTorch) to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[0659] 3. Training the AI ​​model:

[0660] The server uses the analyzed results to train an AI model that learns and mimics the characteristic phrases and speaking styles of the deceased.

[0661] 4. Speech synthesis:

[0662] The text generated by the trained AI model is converted into audio data using a speech synthesis engine (e.g., Amazon Polly), which recreates the voice and speaking style of the deceased.

[0663] 5. User interaction interface:

[0664] Provide an interface that allows users to interact with the deceased within a virtual environment (e.g., a virtual store built using Unity), including a chat window and voice input functionality.

[0665] Users are recommended products and information through simulated dialogue with the deceased.

[0666] Specific examples

[0667] For example, if a user asks "What are the features of this product?" in a virtual store, this text input goes through the following process:

[0668] 1. The user types, "What are the features of this product?"

[0669] The terminal receives this text and sends it to the server.

[0670] 2. The server receives the text and generates a response using an AI model.

[0671] Based on the training data of the deceased, the AI ​​model generates a response such as, "This product was created by founder XXX and is particularly rated for XXX. We believe it will be a perfect fit for your needs."

[0672] 3. The server converts the generated text response into audio data.

[0673] The server uses a speech synthesis engine to create voice data that says, "This product was created by our founder, XXX, and is particularly well-received for XXX. We believe it will perfectly meet your needs."

[0674] 4. The server sends the audio data to the terminal, which then plays the received audio data and delivers it to the user.

[0675] This allows users to have the experience of interacting with the deceased. This system allows the memory of the deceased to live on in the real world as a voice, providing users with a deeply moving experience.

[0676] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0677] Step 1:

[0678] A user uploads the deceased person's voice and text data into the system.

[0679] Input: Audio data (such as call records) and text data (such as social media posts) of the deceased.

[0680] Processing: The device receives these data and sends them to the server using a secure transmission protocol.

[0681] Output: Audio and text data received by the server.

[0682] Step 2:

[0683] The voice data received by the server is converted into text data using an automatic speech recognition (ASR) engine.

[0684] Input: Audio data uploaded to the server.

[0685] Processing: Convert the audio data into text using an ASR engine (e.g., Google Cloud Speech-to-Text).

[0686] Output: Text data.

[0687] Step 3:

[0688] The server uses the text data and analyzes the data using natural language processing (NLP) techniques.

[0689] Input: Text data converted from audio data.

[0690] Processing: The server uses NLP techniques (e.g., TensorFlow or PyTorch) to perform topic and stylistic analysis and extract features of the deceased's phrasing.

[0691] Output: Analysis results and characteristic phraseology data of the deceased.

[0692] Step 4:

[0693] The server trains the AI ​​model based on the analyzed results.

[0694] Input: Data obtained from NLP analysis.

[0695] Processing: The server trains the AI ​​model to learn the speech patterns and expressions of the deceased.

[0696] Output: The trained AI model.

[0697] Step 5:

[0698] The text generated using the trained AI model is converted into audio data.

[0699] Input: Input text from the user and the trained AI model.

[0700] Processing: The server uses the AI ​​model to generate response text, which is then converted into audio data using a speech synthesis engine (e.g., Amazon Polly).

[0701] Output: Audio data recreating the characteristics of the deceased.

[0702] Step 6:

[0703] It provides an interaction interface with the user within the virtual environment.

[0704] Input: User input text or voice.

[0705] Processing: The device responds to the user through a chat window or voice interaction interface within the virtual environment (e.g., a virtual store using Unity).

[0706] Output: The interactive experience provided to the user and the deceased's voice response.

[0707] Step 7:

[0708] Products and information are recommended through simulated dialogue between the user and the deceased.

[0709] Input: The user's question or request.

[0710] Processing: The server uses the AI ​​model to generate product recommendations and information based on the user's needs, and provides them in voice or text format.

[0711] Output: Providing recommended products and information, and improving the user's purchasing experience.

[0712] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0713] The present invention is a system that utilizes records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased to realize real-time conversations that reflect the characteristics of the deceased, and further combines it with an emotion engine that recognizes the user's emotions and adjusts responses, providing a more natural and emotionally rich conversational experience. Specific embodiments of the system are described below.

[0714] System Configuration

[0715] 1. Data Collection

[0716] The user uploads the deceased's audio data (e.g., call records) and text data (e.g., social media posts) to the system.

[0717] The terminal receives data provided by the user and transmits it to the server.

[0718] 2. Data Analysis

[0719] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[0720] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[0721] The server uses an emotion engine to recognize the user's emotions from the analyzed text data and voice data.

[0722] 3. Training the AI ​​model

[0723] The server trains the AI ​​model based on the analyzed results.

[0724] The AI ​​model is designed to learn and mimic the deceased person's characteristic phrasing, speaking style, and emotion recognition results.

[0725] 4. Speech Synthesis

[0726] The server uses a trained AI model to convert the generated text into audio data.

[0727] The audio reproduces the voice and speaking style of the deceased and is further customized to the user's emotional state.

[0728] 5. User interaction interface

[0729] The terminal provides an interface for users to interact with each other, including a chat window and a voice input function.

[0730] The user initiates a dialogue with the deceased person through the interface.

[0731] Specific examples

[0732] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[0733] 1. The user types "Hello, Grandpa."

[0734] The terminal receives this text and sends it to the server.

[0735] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[0736] The emotion engine determines whether the user is expressing emotions such as happiness, sadness, or surprise.

[0737] 3. The server generates a response using an AI model based on the user's emotions.

[0738] Based on the training data of the deceased and the emotion recognition results, the AI ​​model generates a response such as, "Hello, grandson. How are you feeling today?"

[0739] 4. The server converts the generated text response into audio data.

[0740] The server uses a speech synthesis engine to create audio data that says, "Hello, grandson. How are you feeling today?"

[0741] 5. The server sends the audio data to the device.

[0742] The terminal plays the received audio data and delivers it to the user.

[0743] This allows the user to have an experience that feels as if they are conversing with the deceased, and by providing responses that correspond to the user's emotions, it is possible to realize a more emotionally rich conversational experience. This system not only keeps the memory of the deceased alive in the real world as a voice, but also provides responses that take the user's emotions into consideration, thereby providing a moving experience for the user.

[0744] The processing flow will be explained below.

[0745] Step 1:

[0746] A user uploads the deceased person's voice and text data into the system.

[0747] The user selects the data from a dedicated application or web interface and presses the upload button.

[0748] The terminal receives the uploaded data and transmits it to the server using a secure transmission protocol (e.g., SSL / TLS).

[0749] Step 2:

[0750] The server validates the received data and stores it in the database.

[0751] The server checks the data for consistency and integrity and ensures that it is in the correct format.

[0752] The server stores the verified data in a database.

[0753] Step 3:

[0754] The server runs the voice data through an automatic speech recognition (ASR) engine and converts it into text data.

[0755] The server passes the voice data to an ASR engine, which converts the voice signal into corresponding text.

[0756] The server temporarily stores the converted text data.

[0757] Step 4:

[0758] The server runs the text data through a natural language processing (NLP) engine for analysis.

[0759] The server passes the text data to an NLP engine for topic analysis, stylistic analysis, and extraction of features of the deceased's phrasing.

[0760] The server stores the analysis results and uses them to train the AI ​​model.

[0761] Step 5:

[0762] The server runs the analyzed text and voice data through an emotion engine to recognize the user's emotions.

[0763] The server uses an emotion engine to determine the user's emotional state (e.g., happy, sad, angry, etc.) from the text and voice.

[0764] The server stores the emotion recognition results and uses them to generate responses.

[0765] Step 6:

[0766] The server trains the AI ​​model.

[0767] The server uses the analysis and emotion recognition results to input training data into the AI ​​model, allowing it to learn the deceased's characteristic speaking style, phrasing, and emotional state.

[0768] The server stores the trained AI model.

[0769] Step 7:

[0770] The server receives user input and generates a response using AI models and emotion engines.

[0771] The terminal transmits text and voice input from the user to the server.

[0772] The server analyzes the received text and voice to recognize the user's emotional state.

[0773] The server uses AI models to generate appropriate responses based on the characteristics of the deceased and the user's emotional state.

[0774] The server temporarily stores the generated text response.

[0775] Step 8:

[0776] The server generates a text response that is converted into audio data.

[0777] The server uses a text-to-speech (TTS) engine to convert the text response into audio data.

[0778] The server temporarily stores the generated voice data.

[0779] Step 9:

[0780] The server sends the audio data to the terminal.

[0781] The server sends the audio data to the terminal and delivers it to the user.

[0782] Step 10:

[0783] The terminal plays the audio data and delivers the response to the user.

[0784] The terminal uses an audio device to play the received audio data and plays it to the user.

[0785] These steps result in a system that recreates the voice and speaking style of the deceased, providing a natural and immersive conversational experience for the user. Furthermore, responses are generated based on the user's emotions, providing a more emotionally rich conversational experience.

[0786] Example 2

[0787] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0788] The present invention aims to solve the technical problem of reproducing the voice and speaking style of the deceased and providing responses that correspond to the user's emotions in a dialogue system that uses recordings of the deceased while they were alive. Conventional technologies have had problems in that it is difficult to provide real-time dialogue that fully reflects the characteristics of the deceased, and responses based on emotions are not provided, resulting in dialogue that is not natural and emotional.

[0789] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0790] In this invention, the server includes a means for identifying emotions from the analyzed data, a means for training an AI model based on the analysis results to learn characteristics of phrasing and speaking style, and a means for converting the generated text into voice data using the trained AI model, thereby making it possible to faithfully reproduce the characteristics of the deceased and generate responses according to the user's emotions in real time.

[0791] "Voice data from before death" refers to audio information recorded by the deceased person while they were alive.

[0792] "Text data" refers to data of sentences or words stored as character information.

[0793] "Upload" refers to the operation in which a user sends data from their own device to a server.

[0794] "Storage" means storing data in a fixed location so that it can be retrieved as needed.

[0795] "Automatic speech recognition (ASR)" is a technology that converts voice data into text data.

[0796] "Natural Language Processing (NLP)" is a technology that processes and analyzes human language using a computer.

[0797] An "emotion engine" is an algorithm or system for identifying a user's emotion from text or voice data.

[0798] An "AI model" is a mathematical model that uses machine learning algorithms to make predictions and classifications from data.

[0799] "Training" is the process by which an AI model learns patterns and features from data.

[0800] "Speech synthesis" is a technology that converts text data into voice data and reproduces the human voice.

[0801] An "interface" is a window or means through which a user and a system can interact with each other.

[0802] A "secure transmission protocol" is a set of rules and methods for securely communicating data.

[0803] "Real-time" refers to immediate response to user input.

[0804] MODE FOR CARRYING OUT THE INVENTION

[0805] This invention is a system that uses records of a deceased person to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased to realize real-time conversations that reflect the characteristics of the deceased, and by combining it with an emotion engine that recognizes the user's emotions and adjusts responses, it provides a more natural and emotionally rich conversational experience.

[0806] System Configuration

[0807] 1. Data Collection

[0808] The user uploads the deceased's voice data (e.g., call records) and text data (e.g., social media posts) to the system. The user selects the data using a smartphone or PC and presses the upload button through the system's interface.

[0809] The device receives the data provided by the user and transmits it to the server. The device transmits the uploaded data to the server over a secure connection.

[0810] 2. Data Analysis

[0811] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text), sends the audio file to a Google Cloud API, and retrieves the returned text data.

[0812] The server uses natural language processing (NLP) techniques (e.g., spaCy, NLTK) to analyze the text data, perform topic analysis, stylistic analysis, and extract features of the deceased's phrasing. For example, it analyzes the text of the deceased's social media posts to extract topics such as "travel," "family," and "work."

[0813] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion from the analyzed text and voice data. For example, it recognizes the text "I'm very happy today" and extracts the emotion of joy.

[0814] 3. Training the AI ​​model

[0815] The server trains an AI model (e.g., GPT-3) based on the analyzed results. The server trains the AI ​​model using the deceased's phrasing and emotion recognition results to learn how the deceased spoke.

[0816] 4. Speech Synthesis

[0817] The server uses a trained AI model to convert the generated text into speech data (e.g., Amazon Polly), and then sends the generated response text to Amazon Polly to obtain an audio file.

[0818] The server sends the audio data to the terminal, which then sends the created audio data to the end user's terminal, where it is played.

[0819] 5. User interaction interface

[0820] The device provides an interface for the user to interact with the deceased, such as a chat window or voice input function (e.g., a web app or mobile app). The user initiates a conversation with the deceased through the interface.

[0821] The user can enter text into the chat window or speak by pressing the voice input button. For example, the user can enter "Hello, Grandpa," and the device will send this to the server.

[0822] Specific examples

[0823] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[0824] 1. The user types "Hello, Grandpa."

[0825] The terminal receives this text and sends it to the server.

[0826] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[0827] The emotion engine determines whether the user is expressing emotions such as happiness, sadness, or surprise.

[0828] 3. The server generates a response using an AI model based on the user's emotions.

[0829] Based on the training data of the deceased and the emotion recognition results, the AI ​​model generates a response such as, "Hello, grandson. How are you feeling today?"

[0830] 4. The server converts the generated text response into audio data.

[0831] The server uses a speech synthesis engine to create audio data that says, "Hello, grandson. How are you feeling today?"

[0832] 5. The server sends the audio data to the terminal, which then plays the received audio data and delivers it to the user.

[0833] This allows the user to have an experience that feels as if they are having a conversation with the deceased. Furthermore, responses are provided that correspond to the user's emotions, making for a more emotionally rich conversation experience.

[0834] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0835] Step 1:

[0836] Users upload audio and text data of the deceased.

[0837] Input: The user selects an audio file (e.g., call recording) or text file (e.g., social media post) of the deceased person from their smartphone or PC and uploads it to the system.

[0838] Specific operation: When the user clicks the upload button, voice and text data is sent to the system.

[0839] Output: Uploaded data is sent from the user's device to the system's device.

[0840] Step 2:

[0841] The terminal receives data provided by the user and transmits it to the server.

[0842] Input: Voice and text data sent from the user's device.

[0843] Specific operation: The device temporarily stores the received data and then transmits it to the server using a secure connection (e.g., HTTPS protocol).

[0844] Output: The audio and text data is sent to the server.

[0845] Step 3:

[0846] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[0847] Input: The audio data sent to the server.

[0848] What happens: The server calls the Google Cloud Speech-to-Text API to convert the audio file into text.

[0849] Output: The audio data is converted to text data.

[0850] Step 4:

[0851] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[0852] Input: Text data converted by the ASR engine.

[0853] How it works: The server uses spaCy and NLTK to analyze text data and identify specific themes, keywords, and writing styles. For example, analyzing the text of a deceased person's social media posts to extract topics such as travel and family.

[0854] Output: Metadata about the topic, stylistic, and phrasing features of the text.

[0855] Step 5:

[0856] The server uses an emotion engine to recognize the user's emotions from the analyzed text data and voice data.

[0857] Input: Text data after NLP processing.

[0858] How it works: The server uses IBM Watson Tone Analyzer to analyze the emotions contained in the text. For example, it recognizes the text "I'm so happy today" and extracts the emotion of joy.

[0859] Output: Emotion recognition results based on text data.

[0860] Step 6:

[0861] The server trains the AI ​​model based on the analyzed results.

[0862] Input: Emotion recognition results and data on the deceased person's characteristic phrasing and speaking style.

[0863] What it does: The server inputs this data into an AI model (e.g., GPT-3) for training.

[0864] Output: The trained AI model.

[0865] Step 7:

[0866] The server uses a trained AI model to convert the generated text into audio data.

[0867] Input: The text response generated by the trained AI model.

[0868] What happens: The server uses the Amazon Polly API to convert text to speech, for example, "Hello, grandson. How are you feeling today?" into an audio file.

[0869] Output: Audio data reproducing the voice of the deceased.

[0870] Step 8:

[0871] The server sends the audio data to the terminal.

[0872] Input: Speech data generated by a speech synthesis engine.

[0873] What it does: The server generates audio data and sends it to the device using a secure connection (e.g. SSL / TLS).

[0874] Output: The received audio data arrives at the device.

[0875] Step 9:

[0876] The terminal provides an interface with which the user can interact.

[0877] Input: Voice and text data sent to the device.

[0878] Specific operation: The device displays a chat window and voice input function, creating an environment for the user to interact.

[0879] Output: An interface with which the user can initiate an interaction.

[0880] Step 10:

[0881] The user initiates a dialogue with the deceased person through the interface.

[0882] Input: Text or speech entered by the user through the interface.

[0883] Specific operation: When the user starts a conversation by entering something like "Hello, Grandpa," the voice and text are sent via the device to the server, where they are processed again.

[0884] Output: Real-time dialogue with the deceased becomes possible.

[0885] (Application example 2)

[0886] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0887] The present invention relates to a system that provides a dialogue experience using the voice and text data of a deceased person. However, with conventional technologies, it has been difficult to faithfully reproduce the voice and phrasing of the deceased, and it has also been difficult to provide an emotionally rich dialogue experience with the user. Furthermore, there are no systems that can provide natural and emotional dialogue in virtual environments, particularly for specific applications such as product explanations and guidance. The present invention aims to solve these problems.

[0888] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0889] In this invention, the server includes means for uploading and storing voice data and text data from the deceased's life, means for converting the voice data into text data and analyzing the data using natural language processing technology, means for training an AI model based on the analysis results and learning the characteristics of phrasing and speaking style, means for converting the generated text into voice data using the trained AI model, means for providing an interface for dialogue with the user, means for explaining and guiding products in a virtual environment, and means for recognizing the user's emotional state and generating a response accordingly. This allows the user to have an experience that feels as if they are conversing with the deceased, and further enables effective product explanations and guidance in a virtual environment.

[0890] "Voice data and text data from the deceased's lifetime" refers to recordings of voices made by the deceased during their lifetime, as well as sentences and messages left in text format.

[0891] "Uploading and storing" refers to a user sending data to a server via a terminal and storing the data for a long period of time.

[0892] "Converting voice data into text data" refers to converting voice information into corresponding text information using voice recognition technology.

[0893] "Natural language processing technology" refers to technology that enables computers to understand and analyze language spoken naturally by humans.

[0894] "Analyzing data" refers to analyzing collected data and extracting meaning and patterns from it.

[0895] "Training an AI model" refers to having an artificial intelligence learn from collected data and improve its capabilities.

[0896] "Phraseology and speaking characteristics" refers to the specific language and expressions used by a particular person.

[0897] "Converting the generated text into voice data" means converting text data into voice using voice synthesis technology.

[0898] An "interface for interacting with a user" is a mechanism that provides input and output means for a user to interact with a system.

[0899] "Explaining and guiding products in a virtual environment" means providing information about products to users in a virtual space.

[0900] "Recognizing the user's emotional state and generating a response accordingly" means analyzing the user's emotions and generating an appropriate response based on the results.

[0901] This invention is a system that uses the voice and text data of the deceased to provide a conversational experience that makes it seem as if the deceased is still alive. This system realizes real-time conversation that reflects the deceased's distinctive voice and speaking style, as well as the user's emotions.

[0902] First, the user uploads the deceased's voice data (such as call records) and text data (such as social media posts) to the system using their device. The uploaded data is then securely stored on the server using a secure transmission protocol.

[0903] The server then converts the received audio data into text using an automatic speech recognition (ASR) engine (e.g., Google Speech-to-Text), which is then analyzed using natural language processing (NLP) techniques (e.g., spaCy) to extract features such as topic, style, and phrasing.

[0904] The server then uses the analysis results to train an AI model (e.g., OpenAI GPT-4) that can mimic the deceased person's characteristic phrasing and speaking style and generate responses that correspond to the user's emotional state.

[0905] The server then uses the trained AI model to convert the generated text into audio data using a speech synthesis engine (e.g., Amazon Polly), which reproduces the voice and speaking style of the deceased and is further customized according to the user's emotions.

[0906] Finally, the device provides a dialogue interface for the user. Specifically, it includes a chat window and voice input functionality, allowing the user to initiate a dialogue with the deceased through the interface. Furthermore, the system is also equipped with the ability to explain and guide products within the virtual environment. When the user asks something, it can analyze their emotions and provide an appropriate response.

[0907] Specific examples

[0908] For example, if a user types "Please describe this product," this text input goes through the following process:

[0909] 1. The user types, "Please explain this product."

[0910] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[0911] 3. The server generates a response using an AI model based on the user's emotions.

[0912] 4. The server converts the generated text response into audio data.

[0913] 5. The server sends the audio data to the device, and the device plays the audio.

[0914] Example prompt sentence:

[0915] User: "Please explain this product"

[0916] To the AI ​​model:

[0917] User sentiment: 'Curiosity'

[0918] Using training data from a deceased person: "This is a product I love. It has the following features..."

[0919] Response generation and speech synthesis: "Amazon Polly"

[0920] This system not only allows users to have the experience of interacting with the deceased, but also enables effective product explanations and guidance within the virtual environment.

[0921] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0922] Step 1:

[0923] The device provides a means for users to upload audio and text data of the deceased. Users upload data, such as call logs and social media posts, to the device, which then securely transmits it to the server. The input data in this process is audio and text data, and the output data is the uploaded data stored on the server.

[0924] Step 2:

[0925] The server converts the uploaded voice data into text data using an automatic speech recognition (ASR) engine (e.g., Google Speech-to-Text). The input is voice data, which is converted into text through the ASR engine. The output is the converted text data.

[0926] Step 3:

[0927] The server analyzes the converted text data using natural language processing (NLP) techniques (e.g., spaCy). This analysis extracts features of topic, style, and phrasing. The input is the text data converted from speech, and the output is the analyzed text data and its features.

[0928] Step 4:

[0929] The server trains an AI model (e.g., OpenAI GPT-4) based on the analysis results. This training creates a model that reproduces the characteristic speech patterns and expressions of the deceased. The input is the analysis results, and the output is the trained AI model.

[0930] Step 5:

[0931] The server generates text responses using a trained AI model during user interaction. It also uses an emotion analysis engine (e.g., IBM Watson Emotion Analysis) to recognize the user's emotional state and tailor the response accordingly. The input is the user's text input and the result of the emotional state analysis, and the output is the generated response text.

[0932] Step 6:

[0933] The server uses a speech synthesis engine (e.g., Amazon Polly) to convert the generated text into the deceased person's voice. The input is the generated text response, and the output is the converted voice data.

[0934] Step 7:

[0935] The server sends the converted voice data to the terminal, and the terminal plays the voice data. The user listens to the played voice through the dialogue interface and makes the following input: The input is the voice data sent from the server, and the output is the voice played to the user.

[0936] Step 8:

[0937] The device provides a dialogue interface that explains and guides users through products in a virtual environment. When a user inputs a question about a product, the server uses emotion analysis and an AI model to generate an appropriate response and transmits it to the device as voice data. The input is the user's question and emotional state, and the output is voice data with the appropriate response.

[0938] Through this detailed process, users can engage in emotionally rich conversations with the deceased and experience product descriptions in a natural way in a virtual store.

[0939] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0940] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0941] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0942] [Third embodiment]

[0943] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0944] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0945] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0946] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0947] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0948] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0949] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0950] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0951] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0952] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0953] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0954] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0955] The present invention is a system that uses records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased and realizes real-time conversation that reflects the characteristics of the deceased. Specific embodiments of the system are described below.

[0956] System Configuration

[0957] 1. Data Collection

[0958] The user uploads the deceased's audio data (e.g., call records) and text data (e.g., social media posts) to the system.

[0959] The terminal receives data provided by the user and transmits it to the server.

[0960] 2. Data Analysis

[0961] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[0962] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[0963] 3. Training the AI ​​model

[0964] The server trains the AI ​​model based on the analyzed results.

[0965] The AI ​​model is designed to learn and mimic the characteristic phrases and speaking styles of the deceased.

[0966] 4. Speech Synthesis

[0967] The server uses a trained AI model to convert the generated text into audio data.

[0968] This audio recreates the voice and speaking style of the deceased.

[0969] 5. User interaction interface

[0970] The terminal provides an interface for users to interact with each other, including a chat window and a voice input function.

[0971] The user initiates a dialogue with the deceased person through the interface.

[0972] Specific examples

[0973] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[0974] 1. The user types "Hello, Grandpa."

[0975] The terminal receives this text and sends it to the server.

[0976] 2. The server receives the text and generates a response using an AI model.

[0977] Based on training data from the deceased, the AI ​​model generates the response, "Hello, grandson."

[0978] 3. The server converts the generated text response into audio data.

[0979] The server uses a speech synthesis engine to create voice data saying "Hello, my grandson."

[0980] 4. The server sends the audio data to the device.

[0981] The terminal plays the received audio data and delivers it to the user.

[0982] This allows users to have an experience that feels as if they are having a conversation with the deceased. This system is designed to keep the memory of the deceased alive in the real world as a voice, and can provide users with a moving experience.

[0983] The processing flow will be explained below.

[0984] Step 1:

[0985] A user uploads the deceased person's voice and text data into the system.

[0986] The user selects the data from a dedicated application or web interface and presses the upload button.

[0987] The terminal receives the uploaded data and transmits it to the server using a secure transmission protocol (e.g., SSL / TLS).

[0988] Step 2:

[0989] The server validates the received data and stores it in the database.

[0990] The server checks the data for consistency and integrity and ensures that it is in the correct format.

[0991] The server stores the verified data in a database.

[0992] Step 3:

[0993] The server runs the voice data through an automatic speech recognition (ASR) engine and converts it into text data.

[0994] The server passes the voice data to an ASR engine, which converts the voice signal into corresponding text.

[0995] The server temporarily stores the converted text data.

[0996] Step 4:

[0997] The server runs the text data through a natural language processing (NLP) engine for analysis.

[0998] The server passes the text data to an NLP engine for topic analysis, stylistic analysis, and extraction of features of the deceased's phrasing.

[0999] The server stores the analysis results and uses them to train the AI ​​model.

[1000] Step 5:

[1001] The server trains the AI ​​model.

[1002] The server uses the analysis results to input training data into the AI ​​model, allowing it to learn the characteristic speaking style and phrasing of the deceased.

[1003] The server stores the trained AI model.

[1004] Step 6:

[1005] The server receives user input and generates a response using an AI model.

[1006] The terminal transmits the text entered by the user to the server.

[1007] The server inputs the received text into an AI model to generate an appropriate response.

[1008] The server temporarily stores the generated text response.

[1009] Step 7:

[1010] The server generates a text response that is converted into audio data.

[1011] The server uses a text-to-speech (TTS) engine to convert the text response into audio data.

[1012] The server temporarily stores the generated voice data.

[1013] Step 8:

[1014] The server sends the audio data to the terminal.

[1015] The server sends the audio data to the terminal and delivers it to the user.

[1016] Step 9:

[1017] The terminal plays the audio data and delivers the response to the user.

[1018] The terminal uses an audio device to play the received audio data and plays it to the user.

[1019] These steps result in a system that recreates the voice and speaking style of the deceased, providing the user with a natural and immersive interactive experience.

[1020] Example 1

[1021] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1022] While photo albums, videos, and written documents have traditionally been used as a means of reminiscing about the deceased, these are one-way information sources and do not allow for two-way dialogue. Furthermore, there is no way to faithfully reproduce the voice and speaking style of the deceased, creating a need for technology that can provide a realistic conversational experience with the deceased. Furthermore, there has been a lack of technology that can provide a more realistic conversational experience by properly analyzing collected data and learning the specific phrases and speaking styles of the deceased.

[1023] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1024] In this invention, the server includes: means for a user to upload voice data and text data of the deceased to the system and transmit the received data to the server using a secure transmission protocol; means for the server to convert the voice data into text using an automatic speech recognition engine and use natural language processing technology to perform topic analysis, stylistic analysis, and extract phrasing features from the text data; means for training an AI model based on the analysis results and learning the phrasing and speaking characteristics; means for the server to convert the generated text using the trained AI model into voice data and reproduce the voice and speaking style of the deceased; and means for the terminal to provide an interface for dialogue with the user and for the user to start the dialogue, allowing the user to have a real-time dialogue with the deceased as if they were alive.

[1025] "Audio data" is digital data that records what the deceased person said.

[1026] "Text data" refers to written information stored in digital format, such as writings or social media posts by the deceased.

[1027] A "secure transmission protocol" is a communication protocol used to ensure security when sending and receiving data, and a specific example is SSL / TLS.

[1028] A "server" is a computer system that analyzes and processes data received from users.

[1029] An "automatic speech recognition engine" is a computer program that analyzes voice data and converts it into text data.

[1030] "Natural language processing technology" is a technology that analyzes text data and understands and extracts meaning and style.

[1031] "Topic analysis" is the process of extracting major topics or themes from text data.

[1032] "Style analysis" is the process of analyzing the writing style and phrasing of text data and extracting its characteristics.

[1033] An "AI model" is an algorithm that is trained using machine learning techniques to perform a specific task.

[1034] A "voice synthesis means" is a process or technology that takes text data as input and generates a specific voice.

[1035] An "interface" is a means or device for a user to interact with a system, and specifically includes a chat window and a voice input function.

[1036] This invention is a system that uses records of a deceased person to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes voice and text data to realize real-time conversations that reflect the characteristics of the deceased person.

[1037] System Configuration

[1038] Data collection

[1039] First, the user uploads the deceased's audio data (e.g., call records) or text data (e.g., social media posts) to the system. The user provides the data by dragging and dropping the audio or text file into a dedicated interface. The device then receives the data provided by the user and sends it to the server using a secure transmission protocol (e.g., SSL / TLS).

[1040] Data analysis

[1041] The received data is processed by the server. The server converts the audio data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text). The converted text data is then analyzed using natural language processing (NLP) techniques (e.g., TensorFlow or spaCy). This analysis process involves topic analysis of the text data, stylistic analysis, and extraction of features of the deceased's phrasing.

[1042] Training an AI model

[1043] The server then uses the analyzed data to train an AI model, such as OpenAI's GPT-3, which learns the deceased's characteristic phrasing and speaking style. Training is carried out on a server cluster equipped with high-performance GPUs, and training progress is monitored in real time.

[1044] Speech synthesis

[1045] Using the trained AI model, the server converts the generated text into speech data, which is generated using Amazon Polly, for example, and utilizes pre-trained voice characteristics of the deceased person to reproduce a more realistic voice.

[1046] User interaction interface

[1047] The device provides an interface for the user to interact with the deceased, including a chat window and voice input functionality. A form is provided for the user to enter text and a button is provided for recording voice commands. The user can then initiate a dialogue and experience the experience as if they were talking to the deceased person.

[1048] Specific examples

[1049] For example, if the user types "Hello, Grandpa" the following happens:

[1050] 1. The user types "Hello, Grandpa."

[1051] The device receives this text and sends it to the server using the HTTPS protocol.

[1052] 2. The server receives the text and generates a response using an AI model (e.g., GPT-3).

[1053] The AI ​​model generates the response, "Hello, my grandson." This process involves inputting a prompt to the model and selecting an appropriate response based on the output it generates.

[1054] 3. The server converts the generated text response into audio data.

[1055] The server uses Amazon Polly to create a voice message saying "Hello, my grandson." The generated voice message is then encoded into the appropriate format (e.g., MP3).

[1056] 4. The server sends the audio data to the device.

[1057] The device then plays the received audio data and delivers it to the user. The device uses its audio player function to play the audio, providing the user with an experience that makes it seem as if the deceased person is speaking.

[1058] This allows users to have a real-time conversation with the deceased as if they were alive, providing a moving experience. The system is designed to keep the memory of the deceased alive in the real world as their voice.

[1059] Prompt Sentence Examples

[1060] Below are some example prompts to input to a generative AI model:

[1061] User: Hello, Grandpa.

[1062] Deceased AI model: Hello, grandson. How are you today?

[1063] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1064] Step 1: Data collection

[1065] Users upload the audio and text data of the deceased to the system. Users provide data by dragging and dropping audio and text files into a dedicated interface (e.g., a web page or application).

[1066] Input: Audio and text data of the deceased

[1067] Output: Data uploaded to the device

[1068] The terminal receives the data provided by the user and sends it to the server using a secure transmission protocol (e.g. SSL / TLS).

[1069] Input: User-provided voice and text data

[1070] Output: Audio and text data sent to the server

[1071] Step 2: Data analysis

[1072] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text). The server receives the voice data as input and generates text data by performing speech recognition processing.

[1073] Input: Audio data

[1074] Output: Text data generated from the audio data

[1075] The server analyzes the generated text data using natural language processing (NLP) techniques (e.g., TensorFlow and spaCy). This analysis process involves topic and stylistic analysis of the text data to extract characteristics of the deceased's phrasing.

[1076] Input: Text data generated from audio data and text data uploaded by users

[1077] Output: Topic analysis results, stylistic analysis results, and phrasing features of text data

[1078] Step 3: Training the AI ​​model

[1079] The server then trains an AI model based on the analysis results. For example, it uses OpenAI's GPT-3 to learn the deceased's characteristic phrases and speaking style. The server receives the analyzed text data as input and uses it as training data for the AI ​​model.

[1080] Input: Topic analysis results, stylistic analysis results, and phrasing characteristics of text data

[1081] Output: A trained AI model

[1082] Step 4: Text-to-Speech

[1083] The server uses a trained AI model to convert the generated text into speech data, which is generated using Amazon Polly, for example, and uses pre-trained voice characteristics of the deceased person to create a realistic voice.

[1084] Input: Text generated by an AI model

[1085] Output: Generated audio data

[1086] Step 5: User interaction interface

[1087] The device provides an interface for user interaction, including a chat window and voice input functionality. It provides a form for the user to enter text and a button for recording voice commands.

[1088] Input: User request to start a conversation

[1089] Output: The interactive interface presented to the user

[1090] The user initiates a conversation with the deceased through the interface. For example, the user types "Hello, Grandpa" into a chat window, which the device then sends to the server. The server uses an AI model to generate a response, converts the generated text back into audio data, and sends it to the device. The device then plays the received audio data and delivers it to the user.

[1091] Input: Text entered by the user (e.g. "Hello, Grandpa")

[1092] Output: A response played in the deceased person's voice (e.g., "Hello, my grandson.")

[1093] (Application example 1)

[1094] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1095] While there are technologies that can revive the memories and interactive experiences of deceased people in the present day, there are no systems yet that allow users to enjoy these experiences in a virtual environment and receive recommendations for specific information and products. The present invention aims to improve the purchasing experience in a virtual environment by providing users with a personalized interactive experience by providing a conversational experience with a deceased person.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1097] In this invention, the server includes a means for uploading and storing voice and text data from the deceased person's life, a means for converting the voice data into text data and analyzing the data using natural language processing technology, and a means for training an AI model based on the analysis results to learn the characteristics of the deceased person's phrasing and speaking style. This allows the user to recreate the deceased person's memories and conversations and receive information and product recommendations through an interactive experience in a virtual environment.

[1098] "Pre-mortem audio and text data" refers to audio and written records of the deceased person while they were alive.

[1099] "Means for uploading and storing" refers to the ability for users to import data into the system and store it on the server.

[1100] "Converting voice data into text data" refers to the process of converting voice data into text using voice recognition technology.

[1101] "Means of analyzing data using natural language processing technology" refers to a method of using NLP technology to analyze text data and extract information.

[1102] "Training an AI model" refers to the process of using machine learning algorithms to learn from analyzed data.

[1103] "Means of learning phrasing and speaking characteristics" refers to the ability of an AI model to mimic the speaking and expressive styles of a particular individual.

[1104] "Means for converting text into audio data" refers to speech synthesis technology that converts the generated text back into audio format.

[1105] "Means for providing dialogue to a user using a specific speech synthesis engine" refers to a method for responding to a user in real time by voice using a specified speech synthesis tool.

[1106] "Means for providing an interactive experience in a virtual environment" refers to means for allowing a user to interact using a virtual space.

[1107] "Means for recommending products and information" refers to a function that suggests specific products and information based on a dialogue with the user.

[1108] The present invention is a system that uses records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased and realizes real-time conversation that reflects the characteristics of the deceased. Specific embodiments of the system are described below.

[1109] System Configuration

[1110] 1. Data Collection:

[1111] The server provides a means for users to upload audio data (e.g., call logs) and text data (e.g., social media posts) of the deceased to the system, which is sent to the server using a secure transmission protocol.

[1112] The terminal receives data provided by the user and transmits it to the server.

[1113] 2. Data Analysis:

[1114] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text).

[1115] The server uses the text data and utilizes natural language processing (NLP) techniques (e.g., TensorFlow and PyTorch) to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[1116] 3. Training the AI ​​model:

[1117] The server uses the analyzed results to train an AI model that learns and mimics the characteristic phrases and speaking styles of the deceased.

[1118] 4. Speech synthesis:

[1119] The text generated by the trained AI model is converted into audio data using a speech synthesis engine (e.g., Amazon Polly), which recreates the voice and speaking style of the deceased.

[1120] 5. User interaction interface:

[1121] Provide an interface that allows users to interact with the deceased within a virtual environment (e.g., a virtual store built using Unity), including a chat window and voice input functionality.

[1122] Users are recommended products and information through simulated dialogue with the deceased.

[1123] Specific examples

[1124] For example, if a user asks "What are the features of this product?" in a virtual store, this text input goes through the following process:

[1125] 1. The user types, "What are the features of this product?"

[1126] The terminal receives this text and sends it to the server.

[1127] 2. The server receives the text and generates a response using an AI model.

[1128] Based on the training data of the deceased, the AI ​​model generates a response such as, "This product was created by founder XXX and is particularly rated for XXX. We believe it will be a perfect fit for your needs."

[1129] 3. The server converts the generated text response into audio data.

[1130] The server uses a speech synthesis engine to create voice data that says, "This product was created by our founder, XXX, and is particularly well-received for XXX. We believe it will perfectly meet your needs."

[1131] 4. The server sends the audio data to the terminal, which then plays the received audio data and delivers it to the user.

[1132] This allows users to have the experience of interacting with the deceased. This system allows the memory of the deceased to live on in the real world as a voice, providing users with a deeply moving experience.

[1133] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1134] Step 1:

[1135] A user uploads the deceased person's voice and text data into the system.

[1136] Input: Audio data (such as call records) and text data (such as social media posts) of the deceased.

[1137] Processing: The device receives these data and sends them to the server using a secure transmission protocol.

[1138] Output: Audio and text data received by the server.

[1139] Step 2:

[1140] The voice data received by the server is converted into text data using an automatic speech recognition (ASR) engine.

[1141] Input: Audio data uploaded to the server.

[1142] Processing: Convert the audio data into text using an ASR engine (e.g., Google Cloud Speech-to-Text).

[1143] Output: Text data.

[1144] Step 3:

[1145] The server uses the text data and analyzes the data using natural language processing (NLP) techniques.

[1146] Input: Text data converted from audio data.

[1147] Processing: The server uses NLP techniques (e.g., TensorFlow or PyTorch) to perform topic and stylistic analysis and extract features of the deceased's phrasing.

[1148] Output: Analysis results and characteristic phraseology data of the deceased.

[1149] Step 4:

[1150] The server trains the AI ​​model based on the analyzed results.

[1151] Input: Data obtained from NLP analysis.

[1152] Processing: The server trains the AI ​​model to learn the speech patterns and expressions of the deceased.

[1153] Output: The trained AI model.

[1154] Step 5:

[1155] The text generated using the trained AI model is converted into audio data.

[1156] Input: Input text from the user and the trained AI model.

[1157] Processing: The server uses the AI ​​model to generate response text, which is then converted into audio data using a speech synthesis engine (e.g., Amazon Polly).

[1158] Output: Audio data recreating the characteristics of the deceased.

[1159] Step 6:

[1160] It provides an interaction interface with the user within the virtual environment.

[1161] Input: User input text or voice.

[1162] Processing: The device responds to the user through a chat window or voice interaction interface within the virtual environment (e.g., a virtual store using Unity).

[1163] Output: The interactive experience provided to the user and the deceased's voice response.

[1164] Step 7:

[1165] Products and information are recommended through simulated dialogue between the user and the deceased.

[1166] Input: The user's question or request.

[1167] Processing: The server uses the AI ​​model to generate product recommendations and information based on the user's needs, and provides them in voice or text format.

[1168] Output: Providing recommended products and information, and improving the user's purchasing experience.

[1169] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1170] The present invention is a system that utilizes records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased to realize real-time conversations that reflect the characteristics of the deceased, and further combines it with an emotion engine that recognizes the user's emotions and adjusts responses, providing a more natural and emotionally rich conversational experience. Specific embodiments of the system are described below.

[1171] System Configuration

[1172] 1. Data Collection

[1173] The user uploads the deceased's audio data (e.g., call records) and text data (e.g., social media posts) to the system.

[1174] The terminal receives data provided by the user and transmits it to the server.

[1175] 2. Data Analysis

[1176] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[1177] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[1178] The server uses an emotion engine to recognize the user's emotions from the analyzed text data and voice data.

[1179] 3. Training the AI ​​model

[1180] The server trains the AI ​​model based on the analyzed results.

[1181] The AI ​​model is designed to learn and mimic the deceased person's characteristic phrasing, speaking style, and emotion recognition results.

[1182] 4. Speech Synthesis

[1183] The server uses a trained AI model to convert the generated text into audio data.

[1184] The audio reproduces the voice and speaking style of the deceased and is further customized to the user's emotional state.

[1185] 5. User interaction interface

[1186] The terminal provides an interface for users to interact with each other, including a chat window and a voice input function.

[1187] The user initiates a dialogue with the deceased person through the interface.

[1188] Specific examples

[1189] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[1190] 1. The user types "Hello, Grandpa."

[1191] The terminal receives this text and sends it to the server.

[1192] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[1193] The emotion engine determines whether the user is expressing emotions such as happiness, sadness, or surprise.

[1194] 3. The server generates a response using an AI model based on the user's emotions.

[1195] Based on the training data of the deceased and the emotion recognition results, the AI ​​model generates a response such as, "Hello, grandson. How are you feeling today?"

[1196] 4. The server converts the generated text response into audio data.

[1197] The server uses a speech synthesis engine to create audio data that says, "Hello, grandson. How are you feeling today?"

[1198] 5. The server sends the audio data to the device.

[1199] The terminal plays the received audio data and delivers it to the user.

[1200] This allows the user to have an experience that feels as if they are conversing with the deceased, and by providing responses that correspond to the user's emotions, it is possible to realize a more emotionally rich conversational experience. This system not only keeps the memory of the deceased alive in the real world as a voice, but also provides responses that take the user's emotions into consideration, thereby providing a moving experience for the user.

[1201] The processing flow will be explained below.

[1202] Step 1:

[1203] A user uploads the deceased person's voice and text data into the system.

[1204] The user selects the data from a dedicated application or web interface and presses the upload button.

[1205] The terminal receives the uploaded data and transmits it to the server using a secure transmission protocol (e.g., SSL / TLS).

[1206] Step 2:

[1207] The server validates the received data and stores it in the database.

[1208] The server checks the data for consistency and integrity and ensures that it is in the correct format.

[1209] The server stores the verified data in a database.

[1210] Step 3:

[1211] The server runs the voice data through an automatic speech recognition (ASR) engine and converts it into text data.

[1212] The server passes the voice data to an ASR engine, which converts the voice signal into corresponding text.

[1213] The server temporarily stores the converted text data.

[1214] Step 4:

[1215] The server runs the text data through a natural language processing (NLP) engine for analysis.

[1216] The server passes the text data to an NLP engine for topic analysis, stylistic analysis, and extraction of features of the deceased's phrasing.

[1217] The server stores the analysis results and uses them to train the AI ​​model.

[1218] Step 5:

[1219] The server runs the analyzed text and voice data through an emotion engine to recognize the user's emotions.

[1220] The server uses an emotion engine to determine the user's emotional state (e.g., happy, sad, angry, etc.) from the text and voice.

[1221] The server stores the emotion recognition results and uses them to generate responses.

[1222] Step 6:

[1223] The server trains the AI ​​model.

[1224] The server uses the analysis and emotion recognition results to input training data into the AI ​​model, allowing it to learn the deceased's characteristic speaking style, phrasing, and emotional state.

[1225] The server stores the trained AI model.

[1226] Step 7:

[1227] The server receives user input and generates a response using AI models and emotion engines.

[1228] The terminal transmits text and voice input from the user to the server.

[1229] The server analyzes the received text and voice to recognize the user's emotional state.

[1230] The server uses AI models to generate appropriate responses based on the characteristics of the deceased and the user's emotional state.

[1231] The server temporarily stores the generated text response.

[1232] Step 8:

[1233] The server generates a text response that is converted into audio data.

[1234] The server uses a text-to-speech (TTS) engine to convert the text response into audio data.

[1235] The server temporarily stores the generated voice data.

[1236] Step 9:

[1237] The server sends the audio data to the terminal.

[1238] The server sends the audio data to the terminal and delivers it to the user.

[1239] Step 10:

[1240] The terminal plays the audio data and delivers the response to the user.

[1241] The terminal uses an audio device to play the received audio data and plays it to the user.

[1242] These steps result in a system that recreates the voice and speaking style of the deceased, providing a natural and immersive conversational experience for the user. Furthermore, responses are generated based on the user's emotions, providing a more emotionally rich conversational experience.

[1243] Example 2

[1244] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1245] The present invention aims to solve the technical problem of reproducing the voice and speaking style of the deceased and providing responses that correspond to the user's emotions in a dialogue system that uses recordings of the deceased while they were alive. Conventional technologies have had problems in that it is difficult to provide real-time dialogue that fully reflects the characteristics of the deceased, and responses based on emotions are not provided, resulting in dialogue that is not natural and emotional.

[1246] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1247] In this invention, the server includes a means for identifying emotions from the analyzed data, a means for training an AI model based on the analysis results to learn characteristics of phrasing and speaking style, and a means for converting the generated text into voice data using the trained AI model, thereby making it possible to faithfully reproduce the characteristics of the deceased and generate responses according to the user's emotions in real time.

[1248] "Voice data from before death" refers to audio information recorded by the deceased person while they were alive.

[1249] "Text data" refers to data of sentences or words stored as character information.

[1250] "Upload" refers to the operation in which a user sends data from their own device to a server.

[1251] "Storage" means storing data in a fixed location so that it can be retrieved as needed.

[1252] "Automatic speech recognition (ASR)" is a technology that converts voice data into text data.

[1253] "Natural Language Processing (NLP)" is a technology that processes and analyzes human language using a computer.

[1254] An "emotion engine" is an algorithm or system for identifying a user's emotion from text or voice data.

[1255] An "AI model" is a mathematical model that uses machine learning algorithms to make predictions and classifications from data.

[1256] "Training" is the process by which an AI model learns patterns and features from data.

[1257] "Speech synthesis" is a technology that converts text data into voice data and reproduces the human voice.

[1258] An "interface" is a window or means through which a user and a system can interact with each other.

[1259] A "secure transmission protocol" is a set of rules and methods for securely communicating data.

[1260] "Real-time" refers to immediate response to user input.

[1261] MODE FOR CARRYING OUT THE INVENTION

[1262] This invention is a system that uses records of a deceased person to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased to realize real-time conversations that reflect the characteristics of the deceased, and by combining it with an emotion engine that recognizes the user's emotions and adjusts responses, it provides a more natural and emotionally rich conversational experience.

[1263] System Configuration

[1264] 1. Data Collection

[1265] The user uploads the deceased's voice data (e.g., call records) and text data (e.g., social media posts) to the system. The user selects the data using a smartphone or PC and presses the upload button through the system's interface.

[1266] The device receives the data provided by the user and transmits it to the server. The device transmits the uploaded data to the server over a secure connection.

[1267] 2. Data Analysis

[1268] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text), sends the audio file to a Google Cloud API, and retrieves the returned text data.

[1269] The server uses natural language processing (NLP) techniques (e.g., spaCy, NLTK) to analyze the text data, perform topic analysis, stylistic analysis, and extract features of the deceased's phrasing. For example, it analyzes the text of the deceased's social media posts to extract topics such as "travel," "family," and "work."

[1270] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion from the analyzed text and voice data. For example, it recognizes the text "I'm very happy today" and extracts the emotion of joy.

[1271] 3. Training the AI ​​model

[1272] The server trains an AI model (e.g., GPT-3) based on the analyzed results. The server trains the AI ​​model using the deceased's phrasing and emotion recognition results to learn how the deceased spoke.

[1273] 4. Speech Synthesis

[1274] The server uses a trained AI model to convert the generated text into speech data (e.g., Amazon Polly), and then sends the generated response text to Amazon Polly to obtain an audio file.

[1275] The server sends the audio data to the terminal, which then sends the created audio data to the end user's terminal, where it is played.

[1276] 5. User interaction interface

[1277] The device provides an interface for the user to interact with the deceased, such as a chat window or voice input function (e.g., a web app or mobile app). The user initiates a conversation with the deceased through the interface.

[1278] The user can enter text into the chat window or speak by pressing the voice input button. For example, the user can enter "Hello, Grandpa," and the device will send this to the server.

[1279] Specific examples

[1280] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[1281] 1. The user types "Hello, Grandpa."

[1282] The terminal receives this text and sends it to the server.

[1283] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[1284] The emotion engine determines whether the user is expressing emotions such as happiness, sadness, or surprise.

[1285] 3. The server generates a response using an AI model based on the user's emotions.

[1286] Based on the training data of the deceased and the emotion recognition results, the AI ​​model generates a response such as, "Hello, grandson. How are you feeling today?"

[1287] 4. The server converts the generated text response into audio data.

[1288] The server uses a speech synthesis engine to create audio data that says, "Hello, grandson. How are you feeling today?"

[1289] 5. The server sends the audio data to the terminal, which then plays the received audio data and delivers it to the user.

[1290] This allows the user to have an experience that feels as if they are having a conversation with the deceased. Furthermore, responses are provided that correspond to the user's emotions, making for a more emotionally rich conversation experience.

[1291] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1292] Step 1:

[1293] Users upload audio and text data of the deceased.

[1294] Input: The user selects an audio file (e.g., call recording) or text file (e.g., social media post) of the deceased person from their smartphone or PC and uploads it to the system.

[1295] Specific operation: When the user clicks the upload button, voice and text data is sent to the system.

[1296] Output: Uploaded data is sent from the user's device to the system's device.

[1297] Step 2:

[1298] The terminal receives data provided by the user and transmits it to the server.

[1299] Input: Voice and text data sent from the user's device.

[1300] Specific operation: The device temporarily stores the received data and then transmits it to the server using a secure connection (e.g., HTTPS protocol).

[1301] Output: The audio and text data is sent to the server.

[1302] Step 3:

[1303] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[1304] Input: The audio data sent to the server.

[1305] What happens: The server calls the Google Cloud Speech-to-Text API to convert the audio file into text.

[1306] Output: The audio data is converted to text data.

[1307] Step 4:

[1308] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[1309] Input: Text data converted by the ASR engine.

[1310] How it works: The server uses spaCy and NLTK to analyze text data and identify specific themes, keywords, and writing styles. For example, analyzing the text of a deceased person's social media posts to extract topics such as travel and family.

[1311] Output: Metadata about the topic, stylistic, and phrasing features of the text.

[1312] Step 5:

[1313] The server uses an emotion engine to recognize the user's emotions from the analyzed text data and voice data.

[1314] Input: Text data after NLP processing.

[1315] How it works: The server uses IBM Watson Tone Analyzer to analyze the emotions contained in the text. For example, it recognizes the text "I'm so happy today" and extracts the emotion of joy.

[1316] Output: Emotion recognition results based on text data.

[1317] Step 6:

[1318] The server trains the AI ​​model based on the analyzed results.

[1319] Input: Emotion recognition results and data on the deceased person's characteristic phrasing and speaking style.

[1320] What it does: The server inputs this data into an AI model (e.g., GPT-3) for training.

[1321] Output: The trained AI model.

[1322] Step 7:

[1323] The server uses a trained AI model to convert the generated text into audio data.

[1324] Input: The text response generated by the trained AI model.

[1325] What happens: The server uses the Amazon Polly API to convert text to speech, for example, "Hello, grandson. How are you feeling today?" into an audio file.

[1326] Output: Audio data reproducing the voice of the deceased.

[1327] Step 8:

[1328] The server sends the audio data to the terminal.

[1329] Input: Speech data generated by a speech synthesis engine.

[1330] What it does: The server generates audio data and sends it to the device using a secure connection (e.g. SSL / TLS).

[1331] Output: The received audio data arrives at the device.

[1332] Step 9:

[1333] The terminal provides an interface with which the user can interact.

[1334] Input: Voice and text data sent to the device.

[1335] Specific operation: The device displays a chat window and voice input function, creating an environment for the user to interact.

[1336] Output: An interface with which the user can initiate an interaction.

[1337] Step 10:

[1338] The user initiates a dialogue with the deceased person through the interface.

[1339] Input: Text or speech entered by the user through the interface.

[1340] Specific operation: When the user starts a conversation by entering something like "Hello, Grandpa," the voice and text are sent via the device to the server, where they are processed again.

[1341] Output: Real-time dialogue with the deceased becomes possible.

[1342] (Application example 2)

[1343] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1344] The present invention relates to a system that provides a dialogue experience using the voice and text data of a deceased person. However, with conventional technologies, it has been difficult to faithfully reproduce the voice and phrasing of the deceased, and it has also been difficult to provide an emotionally rich dialogue experience with the user. Furthermore, there are no systems that can provide natural and emotional dialogue in virtual environments, particularly for specific applications such as product explanations and guidance. The present invention aims to solve these problems.

[1345] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1346] In this invention, the server includes means for uploading and storing voice data and text data from the deceased's life, means for converting the voice data into text data and analyzing the data using natural language processing technology, means for training an AI model based on the analysis results and learning the characteristics of phrasing and speaking style, means for converting the generated text into voice data using the trained AI model, means for providing an interface for dialogue with the user, means for explaining and guiding products in a virtual environment, and means for recognizing the user's emotional state and generating a response accordingly. This allows the user to have an experience that feels as if they are conversing with the deceased, and further enables effective product explanations and guidance in a virtual environment.

[1347] "Voice data and text data from the deceased's lifetime" refers to recordings of voices made by the deceased during their lifetime, as well as sentences and messages left in text format.

[1348] "Uploading and storing" refers to a user sending data to a server via a terminal and storing the data for a long period of time.

[1349] "Converting voice data into text data" refers to converting voice information into corresponding text information using voice recognition technology.

[1350] "Natural language processing technology" refers to technology that enables computers to understand and analyze language spoken naturally by humans.

[1351] "Analyzing data" refers to analyzing collected data and extracting meaning and patterns from it.

[1352] "Training an AI model" refers to having an artificial intelligence learn from collected data and improve its capabilities.

[1353] "Phraseology and speaking characteristics" refers to the specific language and expressions used by a particular person.

[1354] "Converting the generated text into voice data" means converting text data into voice using voice synthesis technology.

[1355] An "interface for interacting with a user" is a mechanism that provides input and output means for a user to interact with a system.

[1356] "Explaining and guiding products in a virtual environment" means providing information about products to users in a virtual space.

[1357] "Recognizing the user's emotional state and generating a response accordingly" means analyzing the user's emotions and generating an appropriate response based on the results.

[1358] This invention is a system that uses the voice and text data of the deceased to provide a conversational experience that makes it seem as if the deceased is still alive. This system realizes real-time conversation that reflects the deceased's distinctive voice and speaking style, as well as the user's emotions.

[1359] First, the user uploads the deceased's voice data (such as call records) and text data (such as social media posts) to the system using their device. The uploaded data is then securely stored on the server using a secure transmission protocol.

[1360] The server then converts the received audio data into text using an automatic speech recognition (ASR) engine (e.g., Google Speech-to-Text), which is then analyzed using natural language processing (NLP) techniques (e.g., spaCy) to extract features such as topic, style, and phrasing.

[1361] The server then uses the analysis results to train an AI model (e.g., OpenAI GPT-4) that can mimic the deceased person's characteristic phrasing and speaking style and generate responses that correspond to the user's emotional state.

[1362] The server then uses the trained AI model to convert the generated text into audio data using a speech synthesis engine (e.g., Amazon Polly), which reproduces the voice and speaking style of the deceased and is further customized according to the user's emotions.

[1363] Finally, the device provides a dialogue interface for the user. Specifically, it includes a chat window and voice input functionality, allowing the user to initiate a dialogue with the deceased through the interface. Furthermore, the system is also equipped with the ability to explain and guide products within the virtual environment. When the user asks something, it can analyze their emotions and provide an appropriate response.

[1364] Specific examples

[1365] For example, if a user types "Please describe this product," this text input goes through the following process:

[1366] 1. The user types, "Please explain this product."

[1367] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[1368] 3. The server generates a response using an AI model based on the user's emotions.

[1369] 4. The server converts the generated text response into audio data.

[1370] 5. The server sends the audio data to the device, and the device plays the audio.

[1371] Example prompt sentence:

[1372] User: "Please explain this product"

[1373] To the AI ​​model:

[1374] User sentiment: 'Curiosity'

[1375] Using training data from a deceased person: "This is a product I love. It has the following features..."

[1376] Response generation and speech synthesis: "Amazon Polly"

[1377] This system not only allows users to have the experience of interacting with the deceased, but also enables effective product explanations and guidance within the virtual environment.

[1378] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1379] Step 1:

[1380] The device provides a means for users to upload audio and text data of the deceased. Users upload data, such as call logs and social media posts, to the device, which then securely transmits it to the server. The input data in this process is audio and text data, and the output data is the uploaded data stored on the server.

[1381] Step 2:

[1382] The server converts the uploaded voice data into text data using an automatic speech recognition (ASR) engine (e.g., Google Speech-to-Text). The input is voice data, which is converted into text through the ASR engine. The output is the converted text data.

[1383] Step 3:

[1384] The server analyzes the converted text data using natural language processing (NLP) techniques (e.g., spaCy). This analysis extracts features of topic, style, and phrasing. The input is the text data converted from speech, and the output is the analyzed text data and its features.

[1385] Step 4:

[1386] The server trains an AI model (e.g., OpenAI GPT-4) based on the analysis results. This training creates a model that reproduces the characteristic speech patterns and expressions of the deceased. The input is the analysis results, and the output is the trained AI model.

[1387] Step 5:

[1388] The server generates text responses using a trained AI model during user interaction. It also uses an emotion analysis engine (e.g., IBM Watson Emotion Analysis) to recognize the user's emotional state and tailor the response accordingly. The input is the user's text input and the result of the emotional state analysis, and the output is the generated response text.

[1389] Step 6:

[1390] The server uses a speech synthesis engine (e.g., Amazon Polly) to convert the generated text into the deceased person's voice. The input is the generated text response, and the output is the converted voice data.

[1391] Step 7:

[1392] The server sends the converted voice data to the terminal, and the terminal plays the voice data. The user listens to the played voice through the dialogue interface and makes the following input: The input is the voice data sent from the server, and the output is the voice played to the user.

[1393] Step 8:

[1394] The device provides a dialogue interface that explains and guides users through products in a virtual environment. When a user inputs a question about a product, the server uses emotion analysis and an AI model to generate an appropriate response and transmits it to the device as voice data. The input is the user's question and emotional state, and the output is voice data with the appropriate response.

[1395] Through this detailed process, users can engage in emotionally rich conversations with the deceased and experience product descriptions in a natural way in a virtual store.

[1396] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1397] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1398] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1399] [Fourth embodiment]

[1400] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1401] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1402] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1403] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1404] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1405] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1406] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1407] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1408] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1409] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1410] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1411] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1412] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1413] The present invention is a system that uses records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased and realizes real-time conversation that reflects the characteristics of the deceased. Specific embodiments of the system are described below.

[1414] System Configuration

[1415] 1. Data Collection

[1416] The user uploads the deceased's audio data (e.g., call records) and text data (e.g., social media posts) to the system.

[1417] The terminal receives data provided by the user and transmits it to the server.

[1418] 2. Data Analysis

[1419] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[1420] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[1421] 3. Training the AI ​​model

[1422] The server trains the AI ​​model based on the analyzed results.

[1423] The AI ​​model is designed to learn and mimic the characteristic phrases and speaking styles of the deceased.

[1424] 4. Speech Synthesis

[1425] The server uses a trained AI model to convert the generated text into audio data.

[1426] This audio recreates the voice and speaking style of the deceased.

[1427] 5. User interaction interface

[1428] The terminal provides an interface for users to interact with each other, including a chat window and a voice input function.

[1429] The user initiates a dialogue with the deceased person through the interface.

[1430] Specific examples

[1431] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[1432] 1. The user types "Hello, Grandpa."

[1433] The terminal receives this text and sends it to the server.

[1434] 2. The server receives the text and generates a response using an AI model.

[1435] Based on training data from the deceased, the AI ​​model generates the response, "Hello, grandson."

[1436] 3. The server converts the generated text response into audio data.

[1437] The server uses a speech synthesis engine to create voice data saying "Hello, my grandson."

[1438] 4. The server sends the audio data to the device.

[1439] The terminal plays the received audio data and delivers it to the user.

[1440] This allows users to have an experience that feels as if they are having a conversation with the deceased. This system is designed to keep the memory of the deceased alive in the real world as a voice, and can provide users with a moving experience.

[1441] The processing flow will be explained below.

[1442] Step 1:

[1443] A user uploads the deceased person's voice and text data into the system.

[1444] The user selects the data from a dedicated application or web interface and presses the upload button.

[1445] The terminal receives the uploaded data and transmits it to the server using a secure transmission protocol (e.g., SSL / TLS).

[1446] Step 2:

[1447] The server validates the received data and stores it in the database.

[1448] The server checks the data for consistency and integrity and ensures that it is in the correct format.

[1449] The server stores the verified data in a database.

[1450] Step 3:

[1451] The server runs the voice data through an automatic speech recognition (ASR) engine and converts it into text data.

[1452] The server passes the voice data to an ASR engine, which converts the voice signal into corresponding text.

[1453] The server temporarily stores the converted text data.

[1454] Step 4:

[1455] The server runs the text data through a natural language processing (NLP) engine for analysis.

[1456] The server passes the text data to an NLP engine for topic analysis, stylistic analysis, and extraction of features of the deceased's phrasing.

[1457] The server stores the analysis results and uses them to train the AI ​​model.

[1458] Step 5:

[1459] The server trains the AI ​​model.

[1460] The server uses the analysis results to input training data into the AI ​​model, allowing it to learn the characteristic speaking style and phrasing of the deceased.

[1461] The server stores the trained AI model.

[1462] Step 6:

[1463] The server receives user input and generates a response using an AI model.

[1464] The terminal transmits the text entered by the user to the server.

[1465] The server inputs the received text into an AI model to generate an appropriate response.

[1466] The server temporarily stores the generated text response.

[1467] Step 7:

[1468] The server generates a text response that is converted into audio data.

[1469] The server uses a text-to-speech (TTS) engine to convert the text response into audio data.

[1470] The server temporarily stores the generated voice data.

[1471] Step 8:

[1472] The server sends the audio data to the terminal.

[1473] The server sends the audio data to the terminal and delivers it to the user.

[1474] Step 9:

[1475] The terminal plays the audio data and delivers the response to the user.

[1476] The terminal uses an audio device to play the received audio data and plays it to the user.

[1477] These steps result in a system that recreates the voice and speaking style of the deceased, providing the user with a natural and immersive interactive experience.

[1478] Example 1

[1479] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1480] While photo albums, videos, and written documents have traditionally been used as a means of reminiscing about the deceased, these are one-way information sources and do not allow for two-way dialogue. Furthermore, there is no way to faithfully reproduce the voice and speaking style of the deceased, creating a need for technology that can provide a realistic conversational experience with the deceased. Furthermore, there has been a lack of technology that can provide a more realistic conversational experience by properly analyzing collected data and learning the specific phrases and speaking styles of the deceased.

[1481] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1482] In this invention, the server includes: means for a user to upload voice data and text data of the deceased to the system and transmit the received data to the server using a secure transmission protocol; means for the server to convert the voice data into text using an automatic speech recognition engine and use natural language processing technology to perform topic analysis, stylistic analysis, and extract phrasing features from the text data; means for training an AI model based on the analysis results and learning the phrasing and speaking characteristics; means for the server to convert the generated text using the trained AI model into voice data and reproduce the voice and speaking style of the deceased; and means for the terminal to provide an interface for dialogue with the user and for the user to start the dialogue, allowing the user to have a real-time dialogue with the deceased as if they were alive.

[1483] "Audio data" is digital data that records what the deceased person said.

[1484] "Text data" refers to written information stored in digital format, such as writings or social media posts by the deceased.

[1485] A "secure transmission protocol" is a communication protocol used to ensure security when sending and receiving data, and a specific example is SSL / TLS.

[1486] A "server" is a computer system that analyzes and processes data received from users.

[1487] An "automatic speech recognition engine" is a computer program that analyzes voice data and converts it into text data.

[1488] "Natural language processing technology" is a technology that analyzes text data and understands and extracts meaning and style.

[1489] "Topic analysis" is the process of extracting major topics or themes from text data.

[1490] "Style analysis" is the process of analyzing the writing style and phrasing of text data and extracting its characteristics.

[1491] An "AI model" is an algorithm that is trained using machine learning techniques to perform a specific task.

[1492] A "voice synthesis means" is a process or technology that takes text data as input and generates a specific voice.

[1493] An "interface" is a means or device for a user to interact with a system, and specifically includes a chat window and a voice input function.

[1494] This invention is a system that uses records of a deceased person to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes voice and text data to realize real-time conversations that reflect the characteristics of the deceased person.

[1495] System Configuration

[1496] Data collection

[1497] First, the user uploads the deceased's audio data (e.g., call records) or text data (e.g., social media posts) to the system. The user provides the data by dragging and dropping the audio or text file into a dedicated interface. The device then receives the data provided by the user and sends it to the server using a secure transmission protocol (e.g., SSL / TLS).

[1498] Data analysis

[1499] The received data is processed by the server. The server converts the audio data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text). The converted text data is then analyzed using natural language processing (NLP) techniques (e.g., TensorFlow or spaCy). This analysis process involves topic analysis of the text data, stylistic analysis, and extraction of features of the deceased's phrasing.

[1500] Training an AI model

[1501] The server then uses the analyzed data to train an AI model, such as OpenAI's GPT-3, which learns the deceased's characteristic phrasing and speaking style. Training is carried out on a server cluster equipped with high-performance GPUs, and training progress is monitored in real time.

[1502] Speech synthesis

[1503] Using the trained AI model, the server converts the generated text into speech data, which is generated using Amazon Polly, for example, and utilizes pre-trained voice characteristics of the deceased person to reproduce a more realistic voice.

[1504] User interaction interface

[1505] The device provides an interface for the user to interact with the deceased, including a chat window and voice input functionality. A form is provided for the user to enter text and a button is provided for recording voice commands. The user can then initiate a dialogue and experience the experience as if they were talking to the deceased person.

[1506] Specific examples

[1507] For example, if the user types "Hello, Grandpa" the following happens:

[1508] 1. The user types "Hello, Grandpa."

[1509] The device receives this text and sends it to the server using the HTTPS protocol.

[1510] 2. The server receives the text and generates a response using an AI model (e.g., GPT-3).

[1511] The AI ​​model generates the response, "Hello, my grandson." This process involves inputting a prompt to the model and selecting an appropriate response based on the output it generates.

[1512] 3. The server converts the generated text response into audio data.

[1513] The server uses Amazon Polly to create a voice message saying "Hello, my grandson." The generated voice message is then encoded into the appropriate format (e.g., MP3).

[1514] 4. The server sends the audio data to the device.

[1515] The device then plays the received audio data and delivers it to the user. The device uses its audio player function to play the audio, providing the user with an experience that makes it seem as if the deceased person is speaking.

[1516] This allows users to have a real-time conversation with the deceased as if they were alive, providing a moving experience. The system is designed to keep the memory of the deceased alive in the real world as their voice.

[1517] Prompt Sentence Examples

[1518] Below are some example prompts to input to a generative AI model:

[1519] User: Hello, Grandpa.

[1520] Deceased AI model: Hello, grandson. How are you today?

[1521] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1522] Step 1: Data collection

[1523] Users upload the audio and text data of the deceased to the system. Users provide data by dragging and dropping audio and text files into a dedicated interface (e.g., a web page or application).

[1524] Input: Audio and text data of the deceased

[1525] Output: Data uploaded to the device

[1526] The terminal receives the data provided by the user and sends it to the server using a secure transmission protocol (e.g. SSL / TLS).

[1527] Input: User-provided voice and text data

[1528] Output: Audio and text data sent to the server

[1529] Step 2: Data analysis

[1530] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text). The server receives the voice data as input and generates text data by performing speech recognition processing.

[1531] Input: Audio data

[1532] Output: Text data generated from the audio data

[1533] The server analyzes the generated text data using natural language processing (NLP) techniques (e.g., TensorFlow and spaCy). This analysis process involves topic and stylistic analysis of the text data to extract characteristics of the deceased's phrasing.

[1534] Input: Text data generated from audio data and text data uploaded by users

[1535] Output: Topic analysis results, stylistic analysis results, and phrasing features of text data

[1536] Step 3: Training the AI ​​model

[1537] The server then trains an AI model based on the analysis results. For example, it uses OpenAI's GPT-3 to learn the deceased's characteristic phrases and speaking style. The server receives the analyzed text data as input and uses it as training data for the AI ​​model.

[1538] Input: Topic analysis results, stylistic analysis results, and phrasing characteristics of text data

[1539] Output: A trained AI model

[1540] Step 4: Text-to-Speech

[1541] The server uses a trained AI model to convert the generated text into speech data, which is generated using Amazon Polly, for example, and uses pre-trained voice characteristics of the deceased person to create a realistic voice.

[1542] Input: Text generated by an AI model

[1543] Output: Generated audio data

[1544] Step 5: User interaction interface

[1545] The device provides an interface for user interaction, including a chat window and voice input functionality. It provides a form for the user to enter text and a button for recording voice commands.

[1546] Input: User request to start a conversation

[1547] Output: The interactive interface presented to the user

[1548] The user initiates a conversation with the deceased through the interface. For example, the user types "Hello, Grandpa" into a chat window, which the device then sends to the server. The server uses an AI model to generate a response, converts the generated text back into audio data, and sends it to the device. The device then plays the received audio data and delivers it to the user.

[1549] Input: Text entered by the user (e.g. "Hello, Grandpa")

[1550] Output: A response played in the deceased person's voice (e.g., "Hello, my grandson.")

[1551] (Application example 1)

[1552] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1553] While there are technologies that can revive the memories and interactive experiences of deceased people in the present day, there are no systems yet that allow users to enjoy these experiences in a virtual environment and receive recommendations for specific information and products. The present invention aims to improve the purchasing experience in a virtual environment by providing users with a personalized interactive experience by providing a conversational experience with a deceased person.

[1554] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1555] In this invention, the server includes a means for uploading and storing voice and text data from the deceased person's life, a means for converting the voice data into text data and analyzing the data using natural language processing technology, and a means for training an AI model based on the analysis results to learn the characteristics of the deceased person's phrasing and speaking style. This allows the user to recreate the deceased person's memories and conversations and receive information and product recommendations through an interactive experience in a virtual environment.

[1556] "Pre-mortem audio and text data" refers to audio and written records of the deceased person while they were alive.

[1557] "Means for uploading and storing" refers to the ability for users to import data into the system and store it on the server.

[1558] "Converting voice data into text data" refers to the process of converting voice data into text using voice recognition technology.

[1559] "Means of analyzing data using natural language processing technology" refers to a method of using NLP technology to analyze text data and extract information.

[1560] "Training an AI model" refers to the process of using machine learning algorithms to learn from analyzed data.

[1561] "Means of learning phrasing and speaking characteristics" refers to the ability of an AI model to mimic the speaking and expressive styles of a particular individual.

[1562] "Means for converting text into audio data" refers to speech synthesis technology that converts the generated text back into audio format.

[1563] "Means for providing dialogue to a user using a specific speech synthesis engine" refers to a method for responding to a user in real time by voice using a specified speech synthesis tool.

[1564] "Means for providing an interactive experience in a virtual environment" refers to means for allowing a user to interact using a virtual space.

[1565] "Means for recommending products and information" refers to a function that suggests specific products and information based on a dialogue with the user.

[1566] The present invention is a system that uses records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased and realizes real-time conversation that reflects the characteristics of the deceased. Specific embodiments of the system are described below.

[1567] System Configuration

[1568] 1. Data Collection:

[1569] The server provides a means for users to upload audio data (e.g., call logs) and text data (e.g., social media posts) of the deceased to the system, which is sent to the server using a secure transmission protocol.

[1570] The terminal receives data provided by the user and transmits it to the server.

[1571] 2. Data Analysis:

[1572] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text).

[1573] The server uses the text data and utilizes natural language processing (NLP) techniques (e.g., TensorFlow and PyTorch) to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[1574] 3. Training the AI ​​model:

[1575] The server uses the analyzed results to train an AI model that learns and mimics the characteristic phrases and speaking styles of the deceased.

[1576] 4. Speech synthesis:

[1577] The text generated by the trained AI model is converted into audio data using a speech synthesis engine (e.g., Amazon Polly), which recreates the voice and speaking style of the deceased.

[1578] 5. User interaction interface:

[1579] Provide an interface that allows users to interact with the deceased within a virtual environment (e.g., a virtual store built using Unity), including a chat window and voice input functionality.

[1580] Users are recommended products and information through simulated dialogue with the deceased.

[1581] Specific examples

[1582] For example, if a user asks "What are the features of this product?" in a virtual store, this text input goes through the following process:

[1583] 1. The user types, "What are the features of this product?"

[1584] The terminal receives this text and sends it to the server.

[1585] 2. The server receives the text and generates a response using an AI model.

[1586] Based on the training data of the deceased, the AI ​​model generates a response such as, "This product was created by founder XXX and is particularly rated for XXX. We believe it will be a perfect fit for your needs."

[1587] 3. The server converts the generated text response into audio data.

[1588] The server uses a speech synthesis engine to create voice data that says, "This product was created by our founder, XXX, and is particularly well-received for XXX. We believe it will perfectly meet your needs."

[1589] 4. The server sends the audio data to the terminal, which then plays the received audio data and delivers it to the user.

[1590] This allows users to have the experience of interacting with the deceased. This system allows the memory of the deceased to live on in the real world as a voice, providing users with a deeply moving experience.

[1591] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1592] Step 1:

[1593] A user uploads the deceased person's voice and text data into the system.

[1594] Input: Audio data (such as call records) and text data (such as social media posts) of the deceased.

[1595] Processing: The device receives these data and sends them to the server using a secure transmission protocol.

[1596] Output: Audio and text data received by the server.

[1597] Step 2:

[1598] The voice data received by the server is converted into text data using an automatic speech recognition (ASR) engine.

[1599] Input: Audio data uploaded to the server.

[1600] Processing: Convert the audio data into text using an ASR engine (e.g., Google Cloud Speech-to-Text).

[1601] Output: Text data.

[1602] Step 3:

[1603] The server uses the text data and analyzes the data using natural language processing (NLP) techniques.

[1604] Input: Text data converted from audio data.

[1605] Processing: The server uses NLP techniques (e.g., TensorFlow or PyTorch) to perform topic and stylistic analysis and extract features of the deceased's phrasing.

[1606] Output: Analysis results and characteristic phraseology data of the deceased.

[1607] Step 4:

[1608] The server trains the AI ​​model based on the analyzed results.

[1609] Input: Data obtained from NLP analysis.

[1610] Processing: The server trains the AI ​​model to learn the speech patterns and expressions of the deceased.

[1611] Output: The trained AI model.

[1612] Step 5:

[1613] The text generated using the trained AI model is converted into audio data.

[1614] Input: Input text from the user and the trained AI model.

[1615] Processing: The server uses the AI ​​model to generate response text, which is then converted into audio data using a speech synthesis engine (e.g., Amazon Polly).

[1616] Output: Audio data recreating the characteristics of the deceased.

[1617] Step 6:

[1618] It provides an interaction interface with the user within the virtual environment.

[1619] Input: User input text or voice.

[1620] Processing: The device responds to the user through a chat window or voice interaction interface within the virtual environment (e.g., a virtual store using Unity).

[1621] Output: The interactive experience provided to the user and the deceased's voice response.

[1622] Step 7:

[1623] Products and information are recommended through simulated dialogue between the user and the deceased.

[1624] Input: The user's question or request.

[1625] Processing: The server uses the AI ​​model to generate product recommendations and information based on the user's needs, and provides them in voice or text format.

[1626] Output: Providing recommended products and information, and improving the user's purchasing experience.

[1627] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1628] The present invention is a system that utilizes records of a deceased person from their lifetime to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased to realize real-time conversations that reflect the characteristics of the deceased, and further combines it with an emotion engine that recognizes the user's emotions and adjusts responses, providing a more natural and emotionally rich conversational experience. Specific embodiments of the system are described below.

[1629] System Configuration

[1630] 1. Data Collection

[1631] The user uploads the deceased's audio data (e.g., call records) and text data (e.g., social media posts) to the system.

[1632] The terminal receives data provided by the user and transmits it to the server.

[1633] 2. Data Analysis

[1634] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[1635] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[1636] The server uses an emotion engine to recognize the user's emotions from the analyzed text data and voice data.

[1637] 3. Training the AI ​​model

[1638] The server trains the AI ​​model based on the analyzed results.

[1639] The AI ​​model is designed to learn and mimic the deceased person's characteristic phrasing, speaking style, and emotion recognition results.

[1640] 4. Speech Synthesis

[1641] The server uses a trained AI model to convert the generated text into audio data.

[1642] The audio reproduces the voice and speaking style of the deceased and is further customized to the user's emotional state.

[1643] 5. User interaction interface

[1644] The terminal provides an interface for users to interact with each other, including a chat window and a voice input function.

[1645] The user initiates a dialogue with the deceased person through the interface.

[1646] Specific examples

[1647] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[1648] 1. The user types "Hello, Grandpa."

[1649] The terminal receives this text and sends it to the server.

[1650] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[1651] The emotion engine determines whether the user is expressing emotions such as happiness, sadness, or surprise.

[1652] 3. The server generates a response using an AI model based on the user's emotions.

[1653] Based on the training data of the deceased and the emotion recognition results, the AI ​​model generates a response such as, "Hello, grandson. How are you feeling today?"

[1654] 4. The server converts the generated text response into audio data.

[1655] The server uses a speech synthesis engine to create audio data that says, "Hello, grandson. How are you feeling today?"

[1656] 5. The server sends the audio data to the device.

[1657] The terminal plays the received audio data and delivers it to the user.

[1658] This allows the user to have an experience that feels as if they are conversing with the deceased, and by providing responses that correspond to the user's emotions, it is possible to realize a more emotionally rich conversational experience. This system not only keeps the memory of the deceased alive in the real world as a voice, but also provides responses that take the user's emotions into consideration, thereby providing a moving experience for the user.

[1659] The processing flow will be explained below.

[1660] Step 1:

[1661] A user uploads the deceased person's voice and text data into the system.

[1662] The user selects the data from a dedicated application or web interface and presses the upload button.

[1663] The terminal receives the uploaded data and transmits it to the server using a secure transmission protocol (e.g., SSL / TLS).

[1664] Step 2:

[1665] The server validates the received data and stores it in the database.

[1666] The server checks the data for consistency and integrity and ensures that it is in the correct format.

[1667] The server stores the verified data in a database.

[1668] Step 3:

[1669] The server runs the voice data through an automatic speech recognition (ASR) engine and converts it into text data.

[1670] The server passes the voice data to an ASR engine, which converts the voice signal into corresponding text.

[1671] The server temporarily stores the converted text data.

[1672] Step 4:

[1673] The server runs the text data through a natural language processing (NLP) engine for analysis.

[1674] The server passes the text data to an NLP engine for topic analysis, stylistic analysis, and extraction of features of the deceased's phrasing.

[1675] The server stores the analysis results and uses them to train the AI ​​model.

[1676] Step 5:

[1677] The server runs the analyzed text and voice data through an emotion engine to recognize the user's emotions.

[1678] The server uses an emotion engine to determine the user's emotional state (e.g., happy, sad, angry, etc.) from the text and voice.

[1679] The server stores the emotion recognition results and uses them to generate responses.

[1680] Step 6:

[1681] The server trains the AI ​​model.

[1682] The server uses the analysis and emotion recognition results to input training data into the AI ​​model, allowing it to learn the deceased's characteristic speaking style, phrasing, and emotional state.

[1683] The server stores the trained AI model.

[1684] Step 7:

[1685] The server receives user input and generates a response using AI models and emotion engines.

[1686] The terminal transmits text and voice input from the user to the server.

[1687] The server analyzes the received text and voice to recognize the user's emotional state.

[1688] The server uses AI models to generate appropriate responses based on the characteristics of the deceased and the user's emotional state.

[1689] The server temporarily stores the generated text response.

[1690] Step 8:

[1691] The server generates a text response that is converted into audio data.

[1692] The server uses a text-to-speech (TTS) engine to convert the text response into audio data.

[1693] The server temporarily stores the generated voice data.

[1694] Step 9:

[1695] The server sends the audio data to the terminal.

[1696] The server sends the audio data to the terminal and delivers it to the user.

[1697] Step 10:

[1698] The terminal plays the audio data and delivers the response to the user.

[1699] The terminal uses an audio device to play the received audio data and plays it to the user.

[1700] These steps result in a system that recreates the voice and speaking style of the deceased, providing a natural and immersive conversational experience for the user. Furthermore, responses are generated based on the user's emotions, providing a more emotionally rich conversational experience.

[1701] Example 2

[1702] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1703] The present invention aims to solve the technical problem of reproducing the voice and speaking style of the deceased and providing responses that correspond to the user's emotions in a dialogue system that uses recordings of the deceased while they were alive. Conventional technologies have had problems in that it is difficult to provide real-time dialogue that fully reflects the characteristics of the deceased, and responses based on emotions are not provided, resulting in dialogue that is not natural and emotional.

[1704] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1705] In this invention, the server includes a means for identifying emotions from the analyzed data, a means for training an AI model based on the analysis results to learn characteristics of phrasing and speaking style, and a means for converting the generated text into voice data using the trained AI model, thereby making it possible to faithfully reproduce the characteristics of the deceased and generate responses according to the user's emotions in real time.

[1706] "Voice data from before death" refers to audio information recorded by the deceased person while they were alive.

[1707] "Text data" refers to data of sentences or words stored as character information.

[1708] "Upload" refers to the operation in which a user sends data from their own device to a server.

[1709] "Storage" means storing data in a fixed location so that it can be retrieved as needed.

[1710] "Automatic speech recognition (ASR)" is a technology that converts voice data into text data.

[1711] "Natural Language Processing (NLP)" is a technology that processes and analyzes human language using a computer.

[1712] An "emotion engine" is an algorithm or system for identifying a user's emotion from text or voice data.

[1713] An "AI model" is a mathematical model that uses machine learning algorithms to make predictions and classifications from data.

[1714] "Training" is the process by which an AI model learns patterns and features from data.

[1715] "Speech synthesis" is a technology that converts text data into voice data and reproduces the human voice.

[1716] An "interface" is a window or means through which a user and a system can interact with each other.

[1717] A "secure transmission protocol" is a set of rules and methods for securely communicating data.

[1718] "Real-time" refers to immediate response to user input.

[1719] MODE FOR CARRYING OUT THE INVENTION

[1720] This invention is a system that uses records of a deceased person to provide a conversational experience that makes it seem as if the person is still alive. This system analyzes the voice and text data of the deceased to realize real-time conversations that reflect the characteristics of the deceased, and by combining it with an emotion engine that recognizes the user's emotions and adjusts responses, it provides a more natural and emotionally rich conversational experience.

[1721] System Configuration

[1722] 1. Data Collection

[1723] The user uploads the deceased's voice data (e.g., call records) and text data (e.g., social media posts) to the system. The user selects the data using a smartphone or PC and presses the upload button through the system's interface.

[1724] The device receives the data provided by the user and transmits it to the server. The device transmits the uploaded data to the server over a secure connection.

[1725] 2. Data Analysis

[1726] The server converts the received voice data into text using an automatic speech recognition (ASR) engine (e.g., Google Cloud Speech-to-Text), sends the audio file to a Google Cloud API, and retrieves the returned text data.

[1727] The server uses natural language processing (NLP) techniques (e.g., spaCy, NLTK) to analyze the text data, perform topic analysis, stylistic analysis, and extract features of the deceased's phrasing. For example, it analyzes the text of the deceased's social media posts to extract topics such as "travel," "family," and "work."

[1728] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion from the analyzed text and voice data. For example, it recognizes the text "I'm very happy today" and extracts the emotion of joy.

[1729] 3. Training the AI ​​model

[1730] The server trains an AI model (e.g., GPT-3) based on the analyzed results. The server trains the AI ​​model using the deceased's phrasing and emotion recognition results to learn how the deceased spoke.

[1731] 4. Speech Synthesis

[1732] The server uses a trained AI model to convert the generated text into speech data (e.g., Amazon Polly), and then sends the generated response text to Amazon Polly to obtain an audio file.

[1733] The server sends the audio data to the terminal, which then sends the created audio data to the end user's terminal, where it is played.

[1734] 5. User interaction interface

[1735] The device provides an interface for the user to interact with the deceased, such as a chat window or voice input function (e.g., a web app or mobile app). The user initiates a conversation with the deceased through the interface.

[1736] The user can enter text into the chat window or speak by pressing the voice input button. For example, the user can enter "Hello, Grandpa," and the device will send this to the server.

[1737] Specific examples

[1738] For example, if a user types "Hello, Grandpa," this text input goes through the following process:

[1739] 1. The user types "Hello, Grandpa."

[1740] The terminal receives this text and sends it to the server.

[1741] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[1742] The emotion engine determines whether the user is expressing emotions such as happiness, sadness, or surprise.

[1743] 3. The server generates a response using an AI model based on the user's emotions.

[1744] Based on the training data of the deceased and the emotion recognition results, the AI ​​model generates a response such as, "Hello, grandson. How are you feeling today?"

[1745] 4. The server converts the generated text response into audio data.

[1746] The server uses a speech synthesis engine to create audio data that says, "Hello, grandson. How are you feeling today?"

[1747] 5. The server sends the audio data to the terminal, which then plays the received audio data and delivers it to the user.

[1748] This allows the user to have an experience that feels as if they are having a conversation with the deceased. Furthermore, responses are provided that correspond to the user's emotions, making for a more emotionally rich conversation experience.

[1749] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1750] Step 1:

[1751] Users upload audio and text data of the deceased.

[1752] Input: The user selects an audio file (e.g., call recording) or text file (e.g., social media post) of the deceased person from their smartphone or PC and uploads it to the system.

[1753] Specific operation: When the user clicks the upload button, voice and text data is sent to the system.

[1754] Output: Uploaded data is sent from the user's device to the system's device.

[1755] Step 2:

[1756] The terminal receives data provided by the user and transmits it to the server.

[1757] Input: Voice and text data sent from the user's device.

[1758] Specific operation: The device temporarily stores the received data and then transmits it to the server using a secure connection (e.g., HTTPS protocol).

[1759] Output: The audio and text data is sent to the server.

[1760] Step 3:

[1761] The server converts the received voice data into text using an automatic speech recognition (ASR) engine.

[1762] Input: The audio data sent to the server.

[1763] What happens: The server calls the Google Cloud Speech-to-Text API to convert the audio file into text.

[1764] Output: The audio data is converted to text data.

[1765] Step 4:

[1766] The server uses the text data and utilizes natural language processing (NLP) techniques to perform topic analysis of the data, stylistic analysis, and extract features of the deceased's phrasing.

[1767] Input: Text data converted by the ASR engine.

[1768] How it works: The server uses spaCy and NLTK to analyze text data and identify specific themes, keywords, and writing styles. For example, analyzing the text of a deceased person's social media posts to extract topics such as travel and family.

[1769] Output: Metadata about the topic, stylistic, and phrasing features of the text.

[1770] Step 5:

[1771] The server uses an emotion engine to recognize the user's emotions from the analyzed text data and voice data.

[1772] Input: Text data after NLP processing.

[1773] How it works: The server uses IBM Watson Tone Analyzer to analyze the emotions contained in the text. For example, it recognizes the text "I'm so happy today" and extracts the emotion of joy.

[1774] Output: Emotion recognition results based on text data.

[1775] Step 6:

[1776] The server trains the AI ​​model based on the analyzed results.

[1777] Input: Emotion recognition results and data on the deceased person's characteristic phrasing and speaking style.

[1778] What it does: The server inputs this data into an AI model (e.g., GPT-3) for training.

[1779] Output: The trained AI model.

[1780] Step 7:

[1781] The server uses a trained AI model to convert the generated text into audio data.

[1782] Input: The text response generated by the trained AI model.

[1783] What happens: The server uses the Amazon Polly API to convert text to speech, for example, "Hello, grandson. How are you feeling today?" into an audio file.

[1784] Output: Audio data reproducing the voice of the deceased.

[1785] Step 8:

[1786] The server sends the audio data to the terminal.

[1787] Input: Speech data generated by a speech synthesis engine.

[1788] What it does: The server generates audio data and sends it to the device using a secure connection (e.g. SSL / TLS).

[1789] Output: The received audio data arrives at the device.

[1790] Step 9:

[1791] The terminal provides an interface with which the user can interact.

[1792] Input: Voice and text data sent to the device.

[1793] Specific operation: The device displays a chat window and voice input function, creating an environment for the user to interact.

[1794] Output: An interface with which the user can initiate an interaction.

[1795] Step 10:

[1796] The user initiates a dialogue with the deceased person through the interface.

[1797] Input: Text or speech entered by the user through the interface.

[1798] Specific operation: When the user starts a conversation by entering something like "Hello, Grandpa," the voice and text are sent via the device to the server, where they are processed again.

[1799] Output: Real-time dialogue with the deceased becomes possible.

[1800] (Application example 2)

[1801] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1802] The present invention relates to a system that provides a dialogue experience using the voice and text data of a deceased person. However, with conventional technologies, it has been difficult to faithfully reproduce the voice and phrasing of the deceased, and it has also been difficult to provide an emotionally rich dialogue experience with the user. Furthermore, there are no systems that can provide natural and emotional dialogue in virtual environments, particularly for specific applications such as product explanations and guidance. The present invention aims to solve these problems.

[1803] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1804] In this invention, the server includes means for uploading and storing voice data and text data from the deceased's life, means for converting the voice data into text data and analyzing the data using natural language processing technology, means for training an AI model based on the analysis results and learning the characteristics of phrasing and speaking style, means for converting the generated text into voice data using the trained AI model, means for providing an interface for dialogue with the user, means for explaining and guiding products in a virtual environment, and means for recognizing the user's emotional state and generating a response accordingly. This allows the user to have an experience that feels as if they are conversing with the deceased, and further enables effective product explanations and guidance in a virtual environment.

[1805] "Voice data and text data from the deceased's lifetime" refers to recordings of voices made by the deceased during their lifetime, as well as sentences and messages left in text format.

[1806] "Uploading and storing" refers to a user sending data to a server via a terminal and storing the data for a long period of time.

[1807] "Converting voice data into text data" refers to converting voice information into corresponding text information using voice recognition technology.

[1808] "Natural language processing technology" refers to technology that enables computers to understand and analyze language spoken naturally by humans.

[1809] "Analyzing data" refers to analyzing collected data and extracting meaning and patterns from it.

[1810] "Training an AI model" refers to having an artificial intelligence learn from collected data and improve its capabilities.

[1811] "Phraseology and speaking characteristics" refers to the specific language and expressions used by a particular person.

[1812] "Converting the generated text into voice data" means converting text data into voice using voice synthesis technology.

[1813] An "interface for interacting with a user" is a mechanism that provides input and output means for a user to interact with a system.

[1814] "Explaining and guiding products in a virtual environment" means providing information about products to users in a virtual space.

[1815] "Recognizing the user's emotional state and generating a response accordingly" means analyzing the user's emotions and generating an appropriate response based on the results.

[1816] This invention is a system that uses the voice and text data of the deceased to provide a conversational experience that makes it seem as if the deceased is still alive. This system realizes real-time conversation that reflects the deceased's distinctive voice and speaking style, as well as the user's emotions.

[1817] First, the user uploads the deceased's voice data (such as call records) and text data (such as social media posts) to the system using their device. The uploaded data is then securely stored on the server using a secure transmission protocol.

[1818] The server then converts the received audio data into text using an automatic speech recognition (ASR) engine (e.g., Google Speech-to-Text), which is then analyzed using natural language processing (NLP) techniques (e.g., spaCy) to extract features such as topic, style, and phrasing.

[1819] The server then uses the analysis results to train an AI model (e.g., OpenAI GPT-4) that can mimic the deceased person's characteristic phrasing and speaking style and generate responses that correspond to the user's emotional state.

[1820] The server then uses the trained AI model to convert the generated text into audio data using a speech synthesis engine (e.g., Amazon Polly), which reproduces the voice and speaking style of the deceased and is further customized according to the user's emotions.

[1821] Finally, the device provides a dialogue interface for the user. Specifically, it includes a chat window and voice input functionality, allowing the user to initiate a dialogue with the deceased through the interface. Furthermore, the system is also equipped with the ability to explain and guide products within the virtual environment. When the user asks something, it can analyze their emotions and provide an appropriate response.

[1822] Specific examples

[1823] For example, if a user types "Please describe this product," this text input goes through the following process:

[1824] 1. The user types, "Please explain this product."

[1825] 2. The server receives the text and uses an emotion engine to recognize the user's emotion.

[1826] 3. The server generates a response using an AI model based on the user's emotions.

[1827] 4. The server converts the generated text response into audio data.

[1828] 5. The server sends the audio data to the device, and the device plays the audio.

[1829] Example prompt sentence:

[1830] User: "Please explain this product"

[1831] To the AI ​​model:

[1832] User sentiment: 'Curiosity'

[1833] Using training data from a deceased person: "This is a product I love. It has the following features..."

[1834] Response generation and speech synthesis: "Amazon Polly"

[1835] This system not only allows users to have the experience of interacting with the deceased, but also enables effective product explanations and guidance within the virtual environment.

[1836] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1837] Step 1:

[1838] The device provides a means for users to upload audio and text data of the deceased. Users upload data, such as call logs and social media posts, to the device, which then securely transmits it to the server. The input data in this process is audio and text data, and the output data is the uploaded data stored on the server.

[1839] Step 2:

[1840] The server converts the uploaded voice data into text data using an automatic speech recognition (ASR) engine (e.g., Google Speech-to-Text). The input is voice data, which is converted into text through the ASR engine. The output is the converted text data.

[1841] Step 3:

[1842] The server analyzes the converted text data using natural language processing (NLP) techniques (e.g., spaCy). This analysis extracts features of topic, style, and phrasing. The input is the text data converted from speech, and the output is the analyzed text data and its features.

[1843] Step 4:

[1844] The server trains an AI model (e.g., OpenAI GPT-4) based on the analysis results. This training creates a model that reproduces the characteristic speech patterns and expressions of the deceased. The input is the analysis results, and the output is the trained AI model.

[1845] Step 5:

[1846] The server generates text responses using a trained AI model during user interaction. It also uses an emotion analysis engine (e.g., IBM Watson Emotion Analysis) to recognize the user's emotional state and tailor the response accordingly. The input is the user's text input and the result of the emotional state analysis, and the output is the generated response text.

[1847] Step 6:

[1848] The server uses a speech synthesis engine (e.g., Amazon Polly) to convert the generated text into the deceased person's voice. The input is the generated text response, and the output is the converted voice data.

[1849] Step 7:

[1850] The server sends the converted voice data to the terminal, and the terminal plays the voice data. The user listens to the played voice through the dialogue interface and makes the following input: The input is the voice data sent from the server, and the output is the voice played to the user.

[1851] Step 8:

[1852] The device provides a dialogue interface that explains and guides users through products in a virtual environment. When a user inputs a question about a product, the server uses emotion analysis and an AI model to generate an appropriate response and transmits it to the device as voice data. The input is the user's question and emotional state, and the output is voice data with the appropriate response.

[1853] Through this detailed process, users can engage in emotionally rich conversations with the deceased and experience product descriptions in a natural way in a virtual store.

[1854] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1855] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1856] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1857] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1858] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1859] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1860] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1861] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1862] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1863] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1864] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1865] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1866] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1867] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1868] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1869] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1870] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1871] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1872] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1873] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1874] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1875] The following is further disclosed regarding the above embodiment.

[1876] (Claim 1)

[1877] A means for uploading and storing pre-mortem audio and text data;

[1878] A means for converting the voice data into text data and analyzing the data using natural language processing technology;

[1879] A method for training an AI model based on the analysis results to learn the characteristics of phrases and speaking styles, and

[1880] A means for converting the generated text into voice data using the trained AI model;

[1881] means for providing an interface for interacting with a user;

[1882] A system including:

[1883] (Claim 2)

[1884] 10. The system of claim 1, wherein a secure transmission protocol is used when uploading voice and text data.

[1885] (Claim 3)

[1886] 10. The system of claim 1, further comprising means for generating responses and conducting interactions in real time based on user input.

[1887] "Example 1"

[1888] (Claim 1)

[1889] a means for a user to upload voice and text data of the deceased person to the system and transmit the received data to a server using a secure transmission protocol;

[1890] A server converts the voice data into text using an automatic speech recognition engine, and extracts the features of the text data through topic analysis, stylistic analysis, and phrasing using natural language processing technology;

[1891] A method for training an AI model based on the analysis results to learn the characteristics of phrases and speaking styles, and

[1892] A speech synthesis means that converts the generated text into speech data using a trained AI model on the server, and reproduces the voice and speaking style of the deceased;

[1893] a terminal providing an interface for interacting with a user, and a means for the user to initiate the interaction;

[1894] A system including:

[1895] (Claim 2)

[1896] 10. The system of claim 1, wherein a secure transmission protocol is used when uploading voice and text data.

[1897] (Claim 3)

[1898] 10. The system of claim 1, further comprising means for generating responses and conducting interactions in real time based on user input.

[1899] "Application Example 1"

[1900] (Claim 1)

[1901] A means for uploading and storing pre-mortem audio and text data;

[1902] A means for converting the voice data into text data and analyzing the data using natural language processing technology;

[1903] A method for training an AI model based on the analysis results to learn the characteristics of phrases and speaking styles, and

[1904] A means for converting the generated text into voice data using the trained AI model;

[1905] a means for providing a dialogue to a user utilizing a specific speech synthesis engine;

[1906] a means for providing an interactive experience within a virtual environment;

[1907] A means for recommending products and information through simulated dialogue between a user and the deceased;

[1908] A system including:

[1909] (Claim 2)

[1910] 10. The system of claim 1, wherein a secure transmission protocol is used when uploading voice and text data.

[1911] (Claim 3)

[1912] 10. The system of claim 1, further comprising means for generating responses and conducting interactions in real time based on user input.

[1913] "Example 2: Combining Emotion Engines"

[1914] (Claim 1)

[1915] A means for uploading and storing pre-mortem audio and text data;

[1916] A means for converting the voice data into text data and analyzing the data using natural language processing technology;

[1917] means for identifying emotions from the analyzed data;

[1918] A method for training an AI model based on the analysis results to learn the characteristics of phrases and speaking styles, and

[1919] A means for converting the generated text into voice data using the trained AI model;

[1920] means for providing an interface for interacting with a user;

[1921] A system including:

[1922] (Claim 2)

[1923] 10. The system of claim 1, wherein a secure transmission protocol is used when uploading voice and text data.

[1924] (Claim 3)

[1925] 10. The system of claim 1, further comprising means for generating responses and conducting dialogue in real time based on the user's emotion recognition.

[1926] "Application example 2 when combining emotion engines"

[1927] (Claim 1)

[1928] A means for uploading and storing pre-mortem audio and text data;

[1929] A means for converting the voice data into text data and analyzing the data using natural language processing technology;

[1930] A method for training an AI model based on the analysis results to learn the characteristics of phrases and speaking styles, and

[1931] A means for converting the generated text into voice data using the trained AI model;

[1932] means for providing an interface for interacting with a user;

[1933] A means of explaining and guiding products in a virtual environment;

[1934] means for recognizing a user's emotional state and generating a response accordingly;

[1935] A system including:

[1936] (Claim 2)

[1937] 10. The system of claim 1, wherein a secure transmission protocol is used when uploading voice and text data.

[1938] (Claim 3)

[1939] 10. The system of claim 1, further comprising means for generating responses and conducting interactions in real time based on user input. [Explanation of symbols]

[1940] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for uploading and storing pre-mortem audio and text data; A means for converting the voice data into text data and analyzing the data using natural language processing technology; A method for training an AI model based on the analysis results to learn the characteristics of phrases and speaking styles, and A means for converting the generated text into voice data using the trained AI model; means for providing an interface for interacting with a user; A system including:

2. 10. The system of claim 1, wherein a secure transmission protocol is used when uploading voice and text data.

3. 10. The system of claim 1, further comprising means for generating responses and conducting interactions in real time based on user input.

Images