System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting and storing basic information and historical messages of a deceased person; means for preprocessing the collected historical messages to train a natural language processing model; means for receiving an input message from a user and generating a reply using the trained natural language processing model; and means for displaying the generated reply to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] People grieving bereavement lack the means to ease the deep sense of loss and loneliness that comes from not being able to speak with their deceased loved ones again. In such situations, existing grief care methods have difficulty providing a conversational experience that is deeply in touch with the individual's inner self. Furthermore, from the perspective of privacy protection, personal information must be handled safely. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means.

[0006] The system includes means for collecting and storing basic information and message history of the deceased person, means for preprocessing the collected message history to train a natural language processing model, means for receiving an input message from a user and generating a response using the trained natural language processing model, and means for displaying the generated response to the user. The system also includes means for collecting feedback from the user and utilizing it in training the natural language processing model to improve the model, and encryption means for securely storing the collected data.

[0007] "Deceased" means a person who has died.

[0008] "Basic information" refers to basic data about an individual, such as name, date of birth, and address.

[0009] "Message history" refers to a record of previously exchanged messages, which may include text messages, images, audio, etc.

[0010] "Collection" refers to the act of gathering necessary information or data.

[0011] "Storage" refers to recording and retaining collected data.

[0012] "Preprocessing" refers to the process of converting data into a suitable format for analysis and modeling.

[0013] A "natural language processing model" refers to a machine learning model for understanding and generating human language.

[0014] "Training" refers to the process of using data to train a model and improve its performance.

[0015] "User" refers to a person who uses this system.

[0016] An "input message" refers to a message that a user sends to the system.

[0017] "Response" refers to a response message that the system generates and displays to the user.

[0018] "Display" refers to outputting the generated response in a form that is visible to the user.

[0019] "Feedback" refers to the evaluation or opinion that a user gives about the system's response.

[0020] "Encryption" refers to the transformation of content using a specific algorithm to protect the data. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention is a system that provides a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing the user with responses generated by a natural language processing model. The program processing of the system is explained below in order.

[0043] Overall system configuration

[0044] This system mainly consists of a server, a terminal, and a user. The roles of each component are as follows:

[0045] 1. Data Collection

[0046] To start using the system, the user provides basic information about the deceased person (such as name and date of birth) and their LINE message history. As a concrete example, we will assume a situation where the user uploads the deceased person's past LINE messages to their device as a text file.

[0047] The server receives the uploaded data and stores it in a secure database, converts it into a suitable format, and stores it encrypted to protect privacy.

[0048] 2. Data Learning

[0049] The server preprocesses the stored message history to train a natural language processing model. Preprocessing includes, for example, tokenizing the message text and removing unnecessary noise. For example, the server analyzes the grammar and syntax of LINE messages to create a dataset for machine learning.

[0050] The server uses natural language processing techniques to train the model to recognize the deceased person's unique speaking style and expressions. During the training process, patterns are extracted from message history and reflected in the model. For example, the model learns the deceased person's frequently used phrases and unique expressions.

[0051] 3. Generating dialogue with the user

[0052] The user enters a message at the terminal to initiate a dialogue with the system. The user can enter, for example, "How was your day?"

[0053] The terminal sends input from the user to the server. At this time, the terminal adds necessary metadata (e.g., a timestamp) to the input message and sends it to the server. As a concrete example, the terminal sends the user's message "How was your day?" to the server.

[0054] The server generates a response based on the message received from the user, using a trained natural language processing model to generate a response that the deceased person would likely respond with, for example, "I may not have done much today, but I'm always thinking of you."

[0055] The terminal displays the response received from the server to the user, allowing the user to have an experience that feels as if they are interacting with the deceased.

[0056] 4. Ongoing dialogue and feedback

[0057] The user can provide feedback on the system's response, for example by being presented with a button to rate whether the response was appropriate.

[0058] The server improves the natural language processing model based on user feedback. The collected feedback information is used to further train the model. For example, the accuracy of the model can be improved by relearning response patterns where users have given positive feedback.

[0059] This allows users to ease their grief through conversations with the deceased. Data privacy is also ensured, making this a system that users can use with peace of mind.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The user uses the device to enter and upload the deceased person's basic information and LINE message history. Specifically, the user enters the deceased person's name and date of birth into the form, selects a text file of past LINE messages, and presses the send button.

[0063] Step 2:

[0064] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server in a secure manner.

[0065] Step 3:

[0066] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[0067] Step 4:

[0068] The server preprocesses the stored message history, including tokenizing the message text, removing unnecessary data, and performing morphological analysis, so that the formatted data is suitable for training natural language processing models.

[0069] Step 5:

[0070] The server uses the preprocessed data to train a natural language processing model. During the training process, the model learns the deceased person's unique speech patterns and phrasing. Specifically, a recurrent neural network and a transformer model are used to build a language model of the deceased person.

[0071] Step 6:

[0072] A user uses a terminal to enter a message to initiate a conversation with the system, for example, "How was your day?", into an input field on the terminal and presses the send button.

[0073] Step 7:

[0074] The terminal receives an input message from the user and sends it to the server, adding necessary metadata (such as a timestamp or user ID) to the input message and ensuring its reliability.

[0075] Step 8:

[0076] The server analyzes messages received from users and uses a trained natural language processing model to generate appropriate responses, which are naturally crafted based on the linguistic style of the deceased.

[0077] Step 9:

[0078] The server then sends the generated response to the terminal, converting the response message into the required format so that it can be properly displayed to the user.

[0079] Step 10:

[0080] The terminal displays the response received from the server to the user, who can then experience the experience of interacting with the deceased person.

[0081] Step 11:

[0082] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[0083] Step 12:

[0084] The server stores the feedback collected from users and reflects it in improving the natural language processing model, analyzing the feedback data and retraining it to improve the quality of responses.

[0085] These steps allow users to ease their grief through an enhanced interactive experience, while ensuring data security and privacy.

[0086] Example 1

[0087] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0088] Currently, there are systems that provide users who want to reminisce about their deceased loved ones with an experience that makes it seem as if they are having a conversation with the deceased. However, these systems do not adequately address issues such as the security of the information provided by the user, continuous improvement of the model based on user feedback, and learning and reproducing the unique speaking style and expressions of the deceased. This makes it difficult for users to obtain a natural and realistic conversational experience.

[0089] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0090] In this invention, the server includes means for collecting and storing basic information and message history of the deceased, means for preprocessing the collected message history to train a natural language processing model, means for receiving an input message from a user and generating a reply using the trained natural language processing model, means for displaying the generated reply to the user, and means for adding metadata such as a timestamp to the input message from the user and transmitting it to the server. This makes it possible to securely store data, improve the model using feedback, and reproduce the deceased's unique speaking style and expressions, thereby providing the user with a more natural and realistic conversation experience.

[0091] "Basic information about the deceased" refers to the name, date of birth, and other basic information needed to identify the deceased.

[0092] "Message history" refers to records of text messages and chats exchanged by a deceased person during their lifetime, and is primarily used as training data for natural language processing models.

[0093] A "natural language processing model" is a machine learning algorithm designed to understand and generate human language, generating appropriate responses to messages input by users.

[0094] "Tokenization" is a preprocessing technique that divides text data into smaller units such as words or phrases.

[0095] "Preprocessing" refers to a series of processes that remove unnecessary noise from collected data and prepare it in a format suitable for training a natural language processing model.

[0096] A "timestamp" is information indicating the date and time when a specific event occurred, and is added to a message input by a user.

[0097] "Feedback" refers to the user's evaluation or opinion on the system's response, and is data used to improve the model.

[0098] A "trained natural language processing model" is a model that has learned based on the deceased person's message history and is capable of reproducing the deceased person's unique speaking style and phrasing.

[0099] "Encryption" is the process of converting collected data using a specific algorithm so that it cannot be easily analyzed or viewed by third parties.

[0100] A "database" is a system for systematically storing and managing collected data.

[0101] The present invention provides a system for providing a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing a user with responses generated by a natural language processing model. Specific embodiments of the system are described below.

[0102] The hardware required to implement this system is a server, user terminals, and a network connection. The software includes a database management system for data collection and storage, a natural language processing model, and encryption algorithms.

[0103] Data collection and storage

[0104] To begin using the system, users first provide basic information about the deceased (such as name and date of birth) and their message history. This information is primarily in the form of a text file, and users upload it to the system using their device. Specifically, users prepare LINE messages with the deceased as a text file and upload it to their device.

[0105] The device collects the information uploaded by the user and sends it to the server, which stores the received information in a database. At this time, the data is converted into an appropriate format, encrypted, and stored securely.

[0106] Data Preprocessing and Training

[0107] The server pre-processes the stored message history, which includes tokenizing the messages and removing unnecessary noise. For example, the server removes special characters and unnecessary whitespace from the message history and tokenizes it into words.

[0108] The pre-processed data is used to train a natural language processing model to learn the distinctive speech patterns and phrasing of the deceased, which the server uses to build the model and extract patterns and features from the messages.

[0109] User interaction generation

[0110] A user types a message at a terminal to initiate a dialogue with the system. For example, the user types "How was your day?"

[0111] The device sends the message entered by the user, along with metadata such as a timestamp, to the server. The server then analyzes the message and uses a trained natural language processing model to generate a response, such as, "I may not have done much today, but I'm always thinking of you."

[0112] The device displays the generated response to the user, allowing the user to experience as if they were interacting with the deceased.

[0113] Feedback and Improvements

[0114] The user provides feedback on the system's response by providing a button to rate whether the response is appropriate or not.

[0115] The server improves the natural language processing model based on user feedback. By collecting feedback data and retraining the model based on it, it becomes possible to generate more accurate responses.

[0116] Specific examples

[0117] For example, if a user wishes to reminisce about a deceased loved one, they could write a prompt like this:

[0118] "What was your favorite movie?"

[0119] "Tell me about the last trip you took together."

[0120] As the user enters these prompts, the system infers how the deceased would have answered these questions and generates an appropriate response, such as, "My favorite movie was 'Favorite Movie Title.' I always enjoyed watching it with you."

[0121] Through this system, users can enjoy the experience of interacting with the deceased and ease the grief of bereavement.In addition, data privacy is ensured, so users can use the system with peace of mind.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1:

[0124] Data collection

[0125] To start using the system, the user uploads the deceased's basic information (such as name and date of birth) and the message history with the deceased to the device. Specifically, the user provides the LINE messages with the deceased in the form of a text file to the device. The input is the deceased's basic information and message history text file, and the output is the input data saved on the device.

[0126] The terminal reads the uploaded text file. Specifically, the terminal loads the file contents into memory, checks the data format, and then sends the data to the server.

[0127] Step 2:

[0128] Data storage

[0129] The server securely receives the data sent from the device and stores it in a database. At this time, the data is converted into an easy-to-analyze format and encrypted. The input is the basic information and message history of the deceased person sent from the device, and the output is securely stored in the database.

[0130] Specifically, the server stores basic information about the deceased in specific fields, converts message history from plain text to JSON format, and encrypts the data using an encryption algorithm to protect privacy.

[0131] Step 3:

[0132] Data Preprocessing

[0133] The server preprocesses the stored message history, which includes tokenization and removal of unnecessary noise. The input is the message history data retrieved from the database, and the output is the preprocessed dataset.

[0134] Specifically, the server removes special characters and unnecessary spaces from the message history, processes it into a grammatically separable form, tokenizes the data, and splits it into words or phrases. Furthermore, during the noise removal process, it removes things like stop verbs, which are generally considered meaningless.

[0135] Step 4:

[0136] Training a natural language processing model

[0137] The server uses the preprocessed dataset to train a natural language processing model, where the input is the preprocessed dataset and the output is the trained natural language processing model.

[0138] Specifically, the server runs machine learning algorithms to extract patterns and features from the message history, training the model to recognize the unique speech patterns and phrasing of the deceased, fine-tuning specific phrases and unique expressions.

[0139] Step 5:

[0140] User interaction generation

[0141] A user enters a message at a terminal to initiate a conversation with the system, for example, entering a prompt such as "How was your day?" The input is the message from the user, and the output is the message sent to the server.

[0142] The terminal sends the message entered by the user to the server, with metadata such as a timestamp attached to the message. Specifically, the terminal formats the message, adds a timestamp, and sends it to the server over the network.

[0143] The server analyzes the received message and generates a response using a trained natural language processing model. The input is the user's message, and the output is the generated response. For example, the server might generate a response like, "I may not have done much today, but I'm always thinking of you."

[0144] The device receives the generated response and displays it to the user, allowing the user to experience the experience as if they were interacting with the deceased person.

[0145] Step 6:

[0146] Feedback and model improvement

[0147] The user provides feedback on the system's response, for example by clicking a button to rate whether the response was appropriate. The input is the feedback from the user, and the output is the feedback data to the server.

[0148] The terminal transmits the user's feedback to the server, specifically, the terminal collects the feedback information and transmits it together with the metadata to the server.

[0149] The server improves the natural language processing model based on user feedback. The input is the feedback data, and the output is an improved natural language processing model. Specifically, the server uses the feedback to retrain the model and improve the accuracy of responses.

[0150] (Application example 1)

[0151] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0152] Many people seek peace of mind through conversations with their deceased loved ones. However, current technology makes it difficult to provide such experiences in real time. Furthermore, systems that provide such conversational experiences are typically limited to home or personal devices, and are rarely available in commercial facilities or brick-and-mortar stores. This presents a challenge in providing valuable experiences for customers, contributing to attracting more customers and improving customer satisfaction.

[0153] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0154] In this invention, the server includes means for collecting and storing basic information and message history of the deceased, means for preprocessing the collected message history to train a natural language processing model, and means for receiving input messages from users and generating responses using the trained natural language processing model. This allows users to have the experience of interacting with the deceased through an application program installed on a terminal in a physical store. This allows users to enjoy a special conversation experience based on their memories of the deceased in real time at commercial facilities or physical stores, which is expected to increase customer attraction and customer satisfaction.

[0155] "Basic information about the deceased" refers to basic personal information about the deceased, such as name, date of birth, and gender.

[0156] A "message history" is a record of text messages exchanged between the user and the deceased person.

[0157] "Collection" is the process of obtaining the required data from the user and storing that data.

[0158] "Preprocessing" is the process of analyzing raw data and converting it into a format suitable for training a natural language processing model.

[0159] A "natural language processing model" is a machine learning model designed to understand and generate human language.

[0160] "Training" is the process of training a natural language processing model using collected data.

[0161] A "trained natural language processing model" is a natural language processing model that has completed learning through collected data.

[0162] A "user input message" is a text message that a user inputs to the system.

[0163] A "reply" is a response message generated by a natural language processing model in response to an input message.

[0164] "Display" is the act of visually presenting the generated response to the user.

[0165] "Brick-and-mortar terminals" are communication terminals such as displays and robots installed within commercial facilities.

[0166] An "application program" is software that runs on a terminal in a physical store and provides users with an interactive experience.

[0167] "Feedback" refers to the evaluation or opinion that a user gives about the system's response.

[0168] "Encryption" is the process of transforming data using a specific algorithm in order to store it securely.

[0169] This invention is a system that collects basic information and message history of the deceased, and provides an experience where users can interact with the deceased based on this information. The system is mainly composed of a server, a terminal, and a user.

[0170] Server Roles

[0171] The server performs the following main functions:

[0172] 1. Data collection and storage: The user uploads the deceased person's basic information (such as name and date of birth) and message history (LINE and text messages, etc.). The server receives this data, encrypts it, and stores it in a secure database (e.g., PostgreSQL).

[0173] 2. Model Preprocessing and Training: The collected message history is preprocessed for the natural language processing model (e.g., OpenAI's GPT-4) to be used. Preprocessing involves tokenizing the message text and removing noise. The server then uses the preprocessed data to train the natural language processing model to learn the deceased's unique speaking style and phrasing.

[0174] Device Role

[0175] The devices installed in physical stores (e.g. smart displays and robots) have the following functions:

[0176] 1. Receiving a message from the user: A message entered by the user through the terminal is sent to the server. For example, the user enters "Mom, how was your day?"

[0177] 2. Receiving and displaying a response: The generated response received from the server is displayed to the user. For example, the server generates a response such as "I may not have done much today, but I'm always thinking of you" and sends it to the terminal.

[0178] User Roles

[0179] A user uses the system as follows:

[0180] 1. Providing information about the deceased: Provide basic information and message history about the deceased when starting the system.

[0181] 2. Start of dialogue: Enter a message into a terminal in the physical store and receive a response from the server.

[0182] 3. Providing feedback: Providing feedback on the server-generated responses and contributing to the improvement of the system.

[0183] Sample prompt sentence

[0184] Based on the user's message entered on the terminal, the server generates a prompt and creates a response. For example, the following prompt is used:

[0185] User message:

[0186] "Mom, I want to cook your favorite dish. What was your favorite?"

[0187] Example prompt sentence:

[0188] Message History

[0189] User: Mom, I want to cook your favorite dish. What was your favorite?

[0190] Deceased:

[0191] Hardware and software used

[0192] Server: PostgreSQL is used as the database management system, and OpenAI's GPT-4 is used as the natural language processing model.

[0193] Terminals: Smart displays and robots are used to enable real-time interaction with users.

[0194] Secure storage: AES (Advanced Encryption Standard) encryption technology is used as the data encryption method.

[0195] This allows users to enjoy interacting with deceased loved ones even in physical stores, and the quality of the interaction experience can be improved by improving the model based on the feedback provided.

[0196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0197] Step 1:

[0198] The user collects basic information and message history of the deceased person and uploads it to the system. Specifically, the user sends text files and LINE message history to the server via their device. The input is the deceased person's information and message history, and the output is the transmission of this data to the server.

[0199] Step 2:

[0200] The server receives the collected basic information and message history of the deceased and stores it in a secure database. Specifically, after receiving the data, it encrypts it using AES encryption technology and stores it in a PostgreSQL database. The input is the data sent by the user, and the output is stored in the database in encrypted form.

[0201] Step 3:

[0202] The server preprocesses the stored message history by tokenizing the message text and removing unnecessary noise (e.g., emojis and special symbols). The input is the message history retrieved from the database, and the output is the tokenized and preprocessed text data.

[0203] Step 4:

[0204] The server uses the preprocessed data to train a natural language processing model. Specifically, it uses the preprocessed dataset to train OpenAI's GPT-4 model to learn the deceased's unique speaking style and phrasing. The input is the preprocessed text data, and the output is the trained natural language processing model.

[0205] Step 5:

[0206] The user initiates a conversation with the deceased through a terminal in a physical store. Specifically, the user enters a message into the terminal and presses the "send" button. The input is the message the user enters into the terminal, and the output is the message being sent to the server.

[0207] Step 6:

[0208] The terminal sends the input message from the user to the server. Specifically, it adds metadata such as a timestamp and user ID to the input message and sends it to the server via the API. The input is the user's input message and metadata, and the output is that the message is sent to the server.

[0209] Step 7:

[0210] The server generates a response using the input message received from the user, specifically by generating a prompt sentence like the following and feeding it to the GPT-4 model:

[0211] Message History

[0212] User: Mom, I want to cook your favorite dish. What was your favorite?

[0213] Deceased:

[0214] The input is the user's input message and prompt, and the output is the generated response.

[0215] Step 8:

[0216] The server returns the generated response to the terminal. Specifically, it sends a response message to the terminal through the API. The input is the generated response, and the output is the response message returned to the terminal.

[0217] Step 9:

[0218] The terminal displays the response received from the server to the user. Specifically, a smart display or robot conveys the response message to the user visually or audibly. The input is the response message sent from the server, and the output is the response displayed to the user.

[0219] Step 10:

[0220] The user provides feedback on the system-generated response by pressing a button to rate whether the response was appropriate. The input is the user's rating, and the output is that rating is sent to the server.

[0221] Step 11:

[0222] The server collects user feedback and uses it to improve the natural language processing model. Specifically, it analyzes the feedback information and uses it to retrain the model. The input is the user feedback, and the output is an improved natural language processing model.

[0223] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0224] The present invention is a system that provides a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing the user with responses generated by a natural language processing model and an emotion engine. The program processing of the system is explained below step by step.

[0225] Overall system configuration

[0226] This system consists of a server, a terminal, and a user. The role of each component is explained in detail below.

[0227] 1. Data Collection

[0228] To start using the system, users enter and upload basic information about the deceased person and their LINE message history. For example, imagine a situation where a user enters the name and date of birth of the deceased person and uploads a text file of past LINE messages.

[0229] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server in a secure manner.

[0230] 2. Data storage

[0231] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[0232] 3. Data Learning

[0233] The server preprocesses the stored message history and trains a natural language processing model, which includes tokenizing the message text, removing unnecessary data, and morphological analysis. The preprocessed data is used to train the model on the deceased person's unique speaking and phrasing patterns.

[0234] 4. Generating dialogue with the user

[0235] A user uses a terminal to enter a message to initiate a conversation with the system. For example, a user might enter, "How was your day?"

[0236] The terminal sends the input message from the user to the server, with necessary metadata (such as a timestamp and user ID) added to the input message.

[0237] The server analyzes the message received from the user and generates an appropriate response using a trained natural language processing model. The emotion engine is also utilized here. Specifically, it recognizes the emotion in the user's input message and adjusts the tone and content of the response based on that emotion. For example, if the user says, "Today was a tough day," the emotion engine recognizes the "tough" emotion and adjusts the response to be more warmhearted.

[0238] The device displays the response received from the server to the user, allowing the user to experience the experience of interacting with the deceased.

[0239] 5. Ongoing dialogue and feedback

[0240] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[0241] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion engine. It analyzes the feedback data and retrains it to improve the quality of responses.

[0242] 6. Data Security

[0243] The server uses encryption methods for storing and transmitting collected data, which ensures the protection of user privacy.

[0244] Through these steps, users can ease their grief through an advanced dialogue experience. Furthermore, by utilizing an emotion engine, the system can generate responses that are in tune with the user's emotions, enabling more personalized care.

[0245] The processing flow will be explained below.

[0246] Step 1:

[0247] The user uses the device to enter basic information about the deceased person and their past message history and upload it. For example, the user enters the deceased person's name and date of birth, selects the LINE message history as a text file, and presses the send button.

[0248] Step 2:

[0249] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server using a secure communication protocol.

[0250] Step 3:

[0251] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[0252] Step 4:

[0253] The server preprocesses the stored message history, which includes tokenizing the message text and removing unnecessary data, and converts the formatted data into a format suitable for training natural language processing models.

[0254] Step 5:

[0255] The server uses the preprocessed data to train a natural language processing model. During the training process, the model learns the deceased person's unique speech and phrasing patterns. Through this process, the model learns the deceased person's linguistic style.

[0256] Step 6:

[0257] A user inputs a message through a terminal to start a conversation with the system, for example, "How was your day?", and presses the send button.

[0258] Step 7:

[0259] The device sends the input message to the server. The message is accompanied by necessary metadata (such as a timestamp and user ID) and is forwarded to the server while ensuring reliability.

[0260] Step 8:

[0261] The server analyzes messages received from users and uses an emotion engine to recognize the user's emotions. For example, if a user types "I had a hard time today," the emotion engine will recognize "sadness" or "stress" from the user's message.

[0262] Step 9:

[0263] The server uses a natural language processing model to generate an appropriate response based on the emotional data recognized by the emotion engine. The response is tailored to reflect the user's emotions. For example, in response to an input such as "Today was tough," a warm response such as "That must have been really tough. Thank you for talking to me" is generated.

[0264] Step 10:

[0265] The server then sends the generated response to the terminal, converting the response message into the required format so that it can be properly displayed to the user.

[0266] Step 11:

[0267] The device displays the responses received from the server to the user, allowing the user to experience the feeling of having a conversation with the deceased. Furthermore, the responses are emotionally sensitive, providing a deeper interaction experience.

[0268] Step 12:

[0269] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[0270] Step 13:

[0271] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion engine. The feedback data is analyzed and retrained to improve the quality of responses. This effort allows for continuous improvement of the overall system performance and user satisfaction.

[0272] This allows users to ease their grief through a sophisticated dialogue experience, allowing the system to provide responses that are sensitive to the user's emotions, while also ensuring data security and privacy.

[0273] Example 2

[0274] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0275] In modern society, people miss their deceased loved ones, but there are limited means to recreate their memories and messages. However, existing technologies have not yet fully developed systems that provide a dialogue experience with the deceased, making it difficult for users to experience the sensation of "talking" with the deceased. Therefore, new methods are needed to provide users with psychological comfort and healing by maintaining contact with the deceased.

[0276] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0277] In this invention, the server includes means for a user to input and upload basic information about the deceased and their communication history, means for securely processing the input and uploaded data and transferring it to the server, means for decompressing and decrypting the received data and storing it in a database, means for preprocessing the stored communication history, tokenizing and analyzing the data, and training a natural language processing model, means for receiving an input message from the user and generating a response using the trained natural language processing model and an emotion estimation engine, and means for displaying the generated response to the user, thereby enabling the user to have a conversation experience with the deceased.

[0278] "Deceased" refers to someone who has already died.

[0279] "Basic information" refers to the main data that can be used to identify an individual, such as name and date of birth.

[0280] "Communication history" refers to a record of past messages and conversations.

[0281] "User" refers to an individual who uses this system.

[0282] "Input and upload" refers to the act of a user providing data to the system.

[0283] "Secure processing" refers to taking measures such as compression and encryption to handle data safely.

[0284] "Server" refers to a computer system that stores and processes data received from users.

[0285] "Decompressing and decrypting" refers to restoring compressed data to its original form and decrypting encrypted data.

[0286] A "database" refers to a system for organizing and storing data.

[0287] "Preprocessing" refers to processes such as data cleaning and tokenization to make the data suitable for training a model.

[0288] "Tokenization" refers to the process of dividing text data into meaningful units.

[0289] A "natural language processing model" refers to a machine learning algorithm for understanding and generating human language.

[0290] An "emotion estimation engine" refers to an algorithm that analyzes and estimates emotions from input text data.

[0291] "Generating a reply" refers to creating an appropriate response to an input message.

[0292] "Display" refers to outputting the generated response to the user's terminal screen.

[0293] This invention is a system that provides a conversational experience with a deceased person. It collects basic information and communication history of the deceased person and provides the user with responses generated by a natural language processing model and an emotion estimation engine. The system is composed of a server, a terminal, and a user.

[0294] Data Collection and Transmission

[0295] To use the system, users must first enter and upload basic information and communication history of the deceased. Specifically, users enter the name and date of birth of the deceased and upload past communication history (e.g., text file format of messages).

[0296] The terminal receives this data, compresses and encrypts it, and transfers it to the server. For data compression, it uses the zip library, and for encryption, it uses the AES encryption library. The data is sent to the server using a secure communication protocol (for example, HTTPS).

[0297] Receiving and storing data

[0298] The server receives the encrypted data, unpacks and decrypts it using a zip library for unpacking and an AES decryption library for decryption. The server then stores the unpacked and decrypted data in a database (e.g., MySQL).

[0299] Data Learning

[0300] The server preprocesses the stored communication history and trains a natural language processing model, which includes tokenization and removal of unnecessary data using morphological analysis tools (e.g., MeCab), to learn the deceased person's unique speech patterns and phrasing.

[0301] User interaction

[0302] A user uses a terminal to enter a message to initiate a conversation with the system. For example, a user might enter, "How was your day?"

[0303] The terminal sends the input message from the user to the server, with metadata such as a timestamp and user ID attached to the message.

[0304] The server analyzes messages received from users and generates appropriate responses using a trained natural language processing model and an emotion estimation engine. For example, if a user says, "Today was a tough day," the emotion engine recognizes the "toughness" and adjusts the response to be more warm-hearted.

[0305] The device displays the response received from the server to the user, allowing the user to experience the experience of interacting with the deceased.

[0306] Continuous feedback and improvement

[0307] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[0308] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion estimation engine. It analyzes the feedback data and retrains it to improve the quality of responses.

[0309] Data Security

[0310] The server uses encryption methods when storing and transmitting collected data, thereby ensuring user privacy.

[0311] Examples of concrete examples and prompts

[0312] For example, if a user sends a message saying "How was your day?", the server will do the following:

[0313] 1. The user uses a terminal to type, "How was your day?"

[0314] 2. The device sends this message to the server.

[0315] 3. The server analyzes the message and uses an emotion engine to recognize the user's emotion.

[0316] 4. The trained natural language processing model generates a response, for example, "I was thinking of you today, how are you?"

[0317] 5. The server generates a response and sends it to the terminal.

[0318] 6. The terminal displays the reply to the user.

[0319] In this way, users can enjoy the interactive experience of interacting with the deceased, and continuous feedback improves the accuracy of the system.

[0320] Examples of prompts include the following:

[0321] User: How was your day?

[0322] Model: I was thinking of you today, how are you?

[0323] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0324] Step 1:

[0325] The user enters and uploads basic information and communication history of the deceased.

[0326] Input: Basic information of the deceased (e.g. name, date of birth), text file of communication history.

[0327] Specific operation: The user enters the name of the deceased in the name input field, the date of birth of the deceased in the date of birth field, selects the text file of the communication history using the file selection function, and presses the upload button.

[0328] Output: User-entered data sent to the terminal.

[0329] Step 2:

[0330] The terminal receives data entered and uploaded by the user, compresses and encrypts it, and transfers it to the server.

[0331] Input: User-entered data (basic information of the deceased, text file of communication history)

[0332] Specific operation: The terminal compresses the data using a data compression library (e.g., zip library), encrypts the compressed data using an encryption library (e.g., AES encryption), and transfers it to the server using the HTTPS protocol.

[0333] Output: The encrypted and compressed data sent to the server.

[0334] Step 3:

[0335] The server receives the encrypted data, decompresses and decrypts it, and stores it in a database.

[0336] Input: Encrypted and compressed data.

[0337] What happens: The server decrypts the data using an AES decryption library, unpacks it using a zip library, and then stores the unpacked data appropriately in the database system (e.g., MySQL).

[0338] Output: The decompressed and decrypted data is stored in the database.

[0339] Step 4:

[0340] The server preprocesses the stored communication history and trains a natural language processing model.

[0341] Input: The deceased person's communication history stored in a database.

[0342] What it does: The server uses a natural language processing library (e.g., NLTK, spaCy) to tokenize the message text and remove unnecessary data. The preprocessed data is used to train a model to learn the unique speaking style and phrasing of the deceased.

[0343] Output: A trained natural language processing model.

[0344] Step 5:

[0345] The user uses the terminal to enter messages to interact with the system.

[0346] Input: A message typed by the user (e.g., "How was your day?")

[0347] Specific operation: The user enters a message in the input form and presses the send button.

[0348] Output: User input messages sent to the terminal.

[0349] Step 6:

[0350] The terminal sends an input message from the user to the server.

[0351] Input: User input message.

[0352] Specific operation: The terminal adds metadata such as a timestamp and user ID to the input message and sends it to the server using the HTTPS protocol.

[0353] Output: The user's input message and metadata sent to the server.

[0354] Step 7:

[0355] The server analyzes messages received from users and generates appropriate responses using a trained natural language processing model and emotion estimation engine.

[0356] Input: The message and metadata sent by the user.

[0357] How it works: The server analyzes the message content using a natural language processing library and identifies the user's sentiment using a sentiment analysis library (e.g., TextBlob, VADER). A trained generative AI model generates a response, and the sentiment estimation engine adjusts the response.

[0358] Output: The generated reply message.

[0359] Step 8:

[0360] The terminal displays the response received from the server to the user.

[0361] Input: The reply message sent by the server.

[0362] Specific operation: The terminal displays the received reply message on the screen, visually presenting it to the user.

[0363] Output: The reply message that is displayed to the user.

[0364] Step 9:

[0365] The user provides feedback on the system's response.

[0366] Input: User ratings and comments.

[0367] Specific operation: The user enters the response rating and comments in the feedback form and presses the submit button.

[0368] Output: Feedback data sent to the device.

[0369] Step 10:

[0370] The server collects feedback from users and reflects it in improving the natural language processing model and emotion estimation engine.

[0371] Input: Feedback data from users.

[0372] What it does: The server stores the feedback data in a database, analyzes it to identify areas for improvement, and retrains the model.

[0373] Output: Improved natural language processing models and sentiment estimation engines.

[0374] (Application example 2)

[0375] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] Conventional dialogue systems have difficulty providing a dialogue experience with the deceased, and lack emotional care when users interact with the deceased through past message history. Furthermore, there is no function to provide support for products related to the deceased within the virtual store, making it difficult for users to select products that are considerate to the deceased. Our goal is to solve these issues and provide a more personal and emotionally sensitive dialogue experience.

[0377] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0378] In this invention, the server includes means for collecting and storing basic information and communication history of the deceased, means for preprocessing the collected communication history to train a natural language processing model, means for receiving an input message from a user and generating a response using the trained natural language processing model, means for displaying the generated response to the user, means for providing support for inquiries about products related to the deceased in a virtual store, and means for adjusting the tone and content of the response using an emotion engine. This enables the user to purchase products related to the deceased in the virtual store while receiving personal support through a dialogue with the deceased.

[0379] "Deceased" refers to a person who has passed away.

[0380] "Basic information" refers to personal data such as the deceased's name and date of birth.

[0381] "Communication history" refers to past messages and conversation records between the deceased and the user.

[0382] A "natural language processing model" is an algorithm that understands and processes human language, generating responses based on input text.

[0383] "Preprocessing" refers to the process of preparing collected data in a form that is easy to analyze by tokenizing it, removing unnecessary data, performing morphological analysis, etc.

[0384] "User" refers to a person who uses this system to have an interactive experience with a deceased person.

[0385] "Input message" refers to a text message that a user sends to the system.

[0386] "Reply" refers to a response message generated by the system that mimics the speech style and emotions of the deceased.

[0387] "Display means" refers to a method or device for displaying the generated response on the user's terminal.

[0388] A "virtual store" refers to a virtual store space where products and services can be purchased online.

[0389] An "emotion engine" is an algorithm that analyzes emotions from a user's input message and adjusts the tone and content of the response based on those emotions.

[0390] The present invention is a system that allows users to have a conversational experience with a deceased person. It collects and stores basic information and communication history of the deceased, and can train a natural language processing model based on that data. Furthermore, it has the function of providing support for inquiries about products related to the deceased in a virtual store.

[0391] Overall system configuration

[0392] This system is composed of a server, a terminal, and a user, and operates as follows.

[0393] Data collection and storage

[0394] To start using the system, users input and upload basic information about the deceased person and their communication history, such as their name, date of birth, and past message history in text format. The device receives this information, compresses and encrypts the data, and sends it to the server.

[0395] The server decompresses and decrypts the received data and stores it in a database. When storing it, it properly associates the deceased person's basic information and message history with the data and stores it in secure storage, thereby protecting the user's privacy.

[0396] Data preprocessing and model training

[0397] The server preprocesses the stored message history, which includes tokenizing the messages, removing unnecessary data, and morphological analysis. The preprocessed data is then used to train a natural language processing model to learn the deceased person's unique phrasing and speaking patterns.

[0398] User interaction generation and display

[0399] The user uses the device to input a message to initiate a conversation with the system. For example, they could ask, "Did the deceased have any special memories of this accessory?" The device then sends this input message to the server, which then analyzes the message and generates an appropriate response using a trained natural language processing model. The server also uses an emotion engine to recognize emotions in the user's input message and adjust the tone and content of the response based on those emotions.

[0400] The device displays the generated response to the user, allowing the user to experience as if they were having a conversation with the deceased person.

[0401] Support functions within the virtual store

[0402] The system includes a function that provides support for inquiries about products related to the deceased in the virtual store. For example, when a user finds an accessory related to the deceased in the virtual store, they can ask about the deceased's memories and stories associated with the accessory and receive appropriate answers. This allows users to select products through a more personalized experience.

[0403] Software and hardware used

[0404] Server: Used for storing data, preprocessing, training natural language processing models, and generating responses.

[0405] Device: The device (smartphone, tablet, etc.) that a user uses to type and receive messages.

[0406] Natural language processing model: Using OpenAI GPT-3 and other models, responses are generated based on the user's input message.

[0407] Emotion engine: Analyzes the sentiment of a user's message and adjusts the tone and content of responses accordingly.

[0408] Examples of concrete examples and prompts

[0409] For example, if a user is searching for an accessory related to a deceased person in a virtual store, a possible question might be, "Did the deceased have any special memories associated with this accessory?"

[0410] Example prompt sentence:

[0411] How was your day?

[0412] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0413] Step 1:

[0414] Data collection

[0415] Users input and upload basic information and communication history of the deceased person into the device. Specifically, they provide the deceased person's name, date of birth, and past message history as a text file. The device receives this data, compresses and encrypts it, and sends it to the server.

[0416] Input: Deceased person's name, date of birth, message history.

[0417] Output: Compressed and encrypted data.

[0418] Step 2:

[0419] Data storage

[0420] The server receives the data sent from the device, decompresses and decrypts it, and then associates the deceased person's basic information and message history with a database and stores them in a secure storage environment to ensure the data remains private.

[0421] Input: Compressed and encrypted data.

[0422] Output: Basic information and message history of the deceased person stored in the database.

[0423] Step 3:

[0424] Data Preprocessing

[0425] The server preprocesses the stored message history by tokenizing the messages, removing unnecessary data, and performing morphological analysis. This process prepares the data in a format that is easy to analyze.

[0426] Input: The saved message history.

[0427] Output: Preprocessed data.

[0428] Step 4:

[0429] Training the model

[0430] The server uses the preprocessed data to train a natural language processing model, which learns the specific phrasing and speech patterns of the deceased.

[0431] Input: Preprocessed data.

[0432] Output: A trained natural language processing model.

[0433] Step 5:

[0434] Dialogue Generation

[0435] The user inputs a message to the system using the terminal, for example, a question such as "Did the deceased have any special memories about this accessory?" The terminal then sends this input message to the server.

[0436] Input: The user's input message.

[0437] Output: The input message sent to the server.

[0438] Step 6:

[0439] Response Generation

[0440] The server analyzes messages sent by users and generates appropriate responses using a trained natural language processing model, including an emotion engine to analyze the user's emotions and adjust the tone and content of the response.

[0441] Input: The input message sent to the server.

[0442] Output: The generated response.

[0443] Step 7:

[0444] Viewing the response

[0445] The terminal displays the generated response received from the server to the user, thereby allowing the user to have a conversational experience with the deceased person.

[0446] Input: The generated response.

[0447] Output: The response displayed to the user.

[0448] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0449] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0450] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0451] [Second embodiment]

[0452] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0453] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0454] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0455] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0456] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0457] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0458] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0459] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0460] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0461] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0462] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0463] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0464] The present invention is a system that provides a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing the user with responses generated by a natural language processing model. The program processing of the system is explained below in order.

[0465] Overall system configuration

[0466] This system mainly consists of a server, a terminal, and a user. The roles of each component are as follows:

[0467] 1. Data Collection

[0468] To start using the system, the user provides basic information about the deceased person (such as name and date of birth) and their LINE message history. As a concrete example, we will assume a situation where the user uploads the deceased person's past LINE messages to their device as a text file.

[0469] The server receives the uploaded data and stores it in a secure database, converts it into a suitable format, and stores it encrypted to protect privacy.

[0470] 2. Data Learning

[0471] The server preprocesses the stored message history to train a natural language processing model. Preprocessing includes, for example, tokenizing the message text and removing unnecessary noise. For example, the server analyzes the grammar and syntax of LINE messages to create a dataset for machine learning.

[0472] The server uses natural language processing techniques to train the model to recognize the deceased person's unique speaking style and expressions. During the training process, patterns are extracted from message history and reflected in the model. For example, the model learns the deceased person's frequently used phrases and unique expressions.

[0473] 3. Generating dialogue with the user

[0474] The user enters a message at the terminal to initiate a dialogue with the system. The user can enter, for example, "How was your day?"

[0475] The terminal sends input from the user to the server. At this time, the terminal adds necessary metadata (e.g., a timestamp) to the input message and sends it to the server. As a concrete example, the terminal sends the user's message "How was your day?" to the server.

[0476] The server generates a response based on the message received from the user, using a trained natural language processing model to generate a response that the deceased person would likely respond with, for example, "I may not have done much today, but I'm always thinking of you."

[0477] The terminal displays the response received from the server to the user, allowing the user to have an experience that feels as if they are interacting with the deceased.

[0478] 4. Ongoing dialogue and feedback

[0479] The user can provide feedback on the system's response, for example by being presented with a button to rate whether the response was appropriate.

[0480] The server improves the natural language processing model based on user feedback. The collected feedback information is used to further train the model. For example, the accuracy of the model can be improved by relearning response patterns where users have given positive feedback.

[0481] This allows users to ease their grief through conversations with the deceased. Data privacy is also ensured, making this a system that users can use with peace of mind.

[0482] The processing flow will be explained below.

[0483] Step 1:

[0484] The user uses the device to enter and upload the deceased person's basic information and LINE message history. Specifically, the user enters the deceased person's name and date of birth into the form, selects a text file of past LINE messages, and presses the send button.

[0485] Step 2:

[0486] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server in a secure manner.

[0487] Step 3:

[0488] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[0489] Step 4:

[0490] The server preprocesses the stored message history, including tokenizing the message text, removing unnecessary data, and performing morphological analysis, so that the formatted data is suitable for training natural language processing models.

[0491] Step 5:

[0492] The server uses the preprocessed data to train a natural language processing model. During the training process, the model learns the deceased person's unique speech patterns and phrasing. Specifically, a recurrent neural network and a transformer model are used to build a language model of the deceased person.

[0493] Step 6:

[0494] A user uses a terminal to enter a message to initiate a conversation with the system, for example, "How was your day?", into an input field on the terminal and presses the send button.

[0495] Step 7:

[0496] The terminal receives an input message from the user and sends it to the server, adding necessary metadata (such as a timestamp or user ID) to the input message and ensuring its reliability.

[0497] Step 8:

[0498] The server analyzes messages received from users and uses a trained natural language processing model to generate appropriate responses, which are naturally crafted based on the linguistic style of the deceased.

[0499] Step 9:

[0500] The server then sends the generated response to the terminal, converting the response message into the required format so that it can be properly displayed to the user.

[0501] Step 10:

[0502] The terminal displays the response received from the server to the user, who can then experience the experience of interacting with the deceased person.

[0503] Step 11:

[0504] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[0505] Step 12:

[0506] The server stores the feedback collected from users and reflects it in improving the natural language processing model, analyzing the feedback data and retraining it to improve the quality of responses.

[0507] These steps allow users to ease their grief through an enhanced interactive experience, while ensuring data security and privacy.

[0508] Example 1

[0509] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0510] Currently, there are systems that provide users who want to reminisce about their deceased loved ones with an experience that makes it seem as if they are having a conversation with the deceased. However, these systems do not adequately address issues such as the security of the information provided by the user, continuous improvement of the model based on user feedback, and learning and reproducing the unique speaking style and expressions of the deceased. This makes it difficult for users to obtain a natural and realistic conversational experience.

[0511] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0512] In this invention, the server includes means for collecting and storing basic information and message history of the deceased, means for preprocessing the collected message history to train a natural language processing model, means for receiving an input message from a user and generating a reply using the trained natural language processing model, means for displaying the generated reply to the user, and means for adding metadata such as a timestamp to the input message from the user and transmitting it to the server. This makes it possible to securely store data, improve the model using feedback, and reproduce the deceased's unique speaking style and expressions, thereby providing the user with a more natural and realistic conversation experience.

[0513] "Basic information about the deceased" refers to the name, date of birth, and other basic information needed to identify the deceased.

[0514] "Message history" refers to records of text messages and chats exchanged by a deceased person during their lifetime, and is primarily used as training data for natural language processing models.

[0515] A "natural language processing model" is a machine learning algorithm designed to understand and generate human language, generating appropriate responses to messages input by users.

[0516] "Tokenization" is a preprocessing technique that divides text data into smaller units such as words or phrases.

[0517] "Preprocessing" refers to a series of processes that remove unnecessary noise from collected data and prepare it in a format suitable for training a natural language processing model.

[0518] A "timestamp" is information indicating the date and time when a specific event occurred, and is added to a message input by a user.

[0519] "Feedback" refers to the user's evaluation or opinion on the system's response, and is data used to improve the model.

[0520] A "trained natural language processing model" is a model that has learned based on the deceased person's message history and is capable of reproducing the deceased person's unique speaking style and phrasing.

[0521] "Encryption" is the process of converting collected data using a specific algorithm so that it cannot be easily analyzed or viewed by third parties.

[0522] A "database" is a system for systematically storing and managing collected data.

[0523] The present invention provides a system for providing a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing a user with responses generated by a natural language processing model. Specific embodiments of the system are described below.

[0524] The hardware required to implement this system is a server, user terminals, and a network connection. The software includes a database management system for data collection and storage, a natural language processing model, and encryption algorithms.

[0525] Data collection and storage

[0526] To begin using the system, users first provide basic information about the deceased (such as name and date of birth) and their message history. This information is primarily in the form of a text file, and users upload it to the system using their device. Specifically, users prepare LINE messages with the deceased as a text file and upload it to their device.

[0527] The device collects the information uploaded by the user and sends it to the server, which stores the received information in a database. At this time, the data is converted into an appropriate format, encrypted, and stored securely.

[0528] Data Preprocessing and Training

[0529] The server pre-processes the stored message history, which includes tokenizing the messages and removing unnecessary noise. For example, the server removes special characters and unnecessary whitespace from the message history and tokenizes it into words.

[0530] The pre-processed data is used to train a natural language processing model to learn the distinctive speech patterns and phrasing of the deceased, which the server uses to build the model and extract patterns and features from the messages.

[0531] User interaction generation

[0532] A user types a message at a terminal to initiate a dialogue with the system. For example, the user types "How was your day?"

[0533] The device sends the message entered by the user, along with metadata such as a timestamp, to the server. The server then analyzes the message and uses a trained natural language processing model to generate a response, such as, "I may not have done much today, but I'm always thinking of you."

[0534] The device displays the generated response to the user, allowing the user to experience as if they were interacting with the deceased.

[0535] Feedback and Improvements

[0536] The user provides feedback on the system's response by providing a button to rate whether the response is appropriate or not.

[0537] The server improves the natural language processing model based on user feedback. By collecting feedback data and retraining the model based on it, it becomes possible to generate more accurate responses.

[0538] Specific examples

[0539] For example, if a user wishes to reminisce about a deceased loved one, they could write a prompt like this:

[0540] "What was your favorite movie?"

[0541] "Tell me about the last trip you took together."

[0542] As the user enters these prompts, the system infers how the deceased would have answered these questions and generates an appropriate response, such as, "My favorite movie was 'Favorite Movie Title.' I always enjoyed watching it with you."

[0543] Through this system, users can enjoy the experience of interacting with the deceased and ease the grief of bereavement.In addition, data privacy is ensured, so users can use the system with peace of mind.

[0544] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0545] Step 1:

[0546] Data collection

[0547] To start using the system, the user uploads the deceased's basic information (such as name and date of birth) and the message history with the deceased to the device. Specifically, the user provides the LINE messages with the deceased in the form of a text file to the device. The input is the deceased's basic information and message history text file, and the output is the input data saved on the device.

[0548] The terminal reads the uploaded text file. Specifically, the terminal loads the file contents into memory, checks the data format, and then sends the data to the server.

[0549] Step 2:

[0550] Data storage

[0551] The server securely receives the data sent from the device and stores it in a database. At this time, the data is converted into an easy-to-analyze format and encrypted. The input is the basic information and message history of the deceased person sent from the device, and the output is securely stored in the database.

[0552] Specifically, the server stores basic information about the deceased in specific fields, converts message history from plain text to JSON format, and encrypts the data using an encryption algorithm to protect privacy.

[0553] Step 3:

[0554] Data Preprocessing

[0555] The server preprocesses the stored message history, which includes tokenization and removal of unnecessary noise. The input is the message history data retrieved from the database, and the output is the preprocessed dataset.

[0556] Specifically, the server removes special characters and unnecessary spaces from the message history, processes it into a grammatically separable form, tokenizes the data, and splits it into words or phrases. Furthermore, during the noise removal process, it removes things like stop verbs, which are generally considered meaningless.

[0557] Step 4:

[0558] Training a natural language processing model

[0559] The server uses the preprocessed dataset to train a natural language processing model, where the input is the preprocessed dataset and the output is the trained natural language processing model.

[0560] Specifically, the server runs machine learning algorithms to extract patterns and features from the message history, training the model to recognize the unique speech patterns and phrasing of the deceased, fine-tuning specific phrases and unique expressions.

[0561] Step 5:

[0562] User interaction generation

[0563] A user enters a message at a terminal to initiate a conversation with the system, for example, entering a prompt such as "How was your day?" The input is the message from the user, and the output is the message sent to the server.

[0564] The terminal sends the message entered by the user to the server, with metadata such as a timestamp attached to the message. Specifically, the terminal formats the message, adds a timestamp, and sends it to the server over the network.

[0565] The server analyzes the received message and generates a response using a trained natural language processing model. The input is the user's message, and the output is the generated response. For example, the server might generate a response like, "I may not have done much today, but I'm always thinking of you."

[0566] The device receives the generated response and displays it to the user, allowing the user to experience the experience as if they were interacting with the deceased person.

[0567] Step 6:

[0568] Feedback and model improvement

[0569] The user provides feedback on the system's response, for example by clicking a button to rate whether the response was appropriate. The input is the feedback from the user, and the output is the feedback data to the server.

[0570] The terminal transmits the user's feedback to the server, specifically, the terminal collects the feedback information and transmits it together with the metadata to the server.

[0571] The server improves the natural language processing model based on user feedback. The input is the feedback data, and the output is an improved natural language processing model. Specifically, the server uses the feedback to retrain the model and improve the accuracy of responses.

[0572] (Application example 1)

[0573] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0574] Many people seek peace of mind through conversations with their deceased loved ones. However, current technology makes it difficult to provide such experiences in real time. Furthermore, systems that provide such conversational experiences are typically limited to home or personal devices, and are rarely available in commercial facilities or brick-and-mortar stores. This presents a challenge in providing valuable experiences for customers, contributing to attracting more customers and improving customer satisfaction.

[0575] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0576] In this invention, the server includes means for collecting and storing basic information and message history of the deceased, means for preprocessing the collected message history to train a natural language processing model, and means for receiving input messages from users and generating responses using the trained natural language processing model. This allows users to have the experience of interacting with the deceased through an application program installed on a terminal in a physical store. This allows users to enjoy a special conversation experience based on their memories of the deceased in real time at commercial facilities or physical stores, which is expected to increase customer attraction and customer satisfaction.

[0577] "Basic information about the deceased" refers to basic personal information about the deceased, such as name, date of birth, and gender.

[0578] A "message history" is a record of text messages exchanged between the user and the deceased person.

[0579] "Collection" is the process of obtaining the required data from the user and storing that data.

[0580] "Preprocessing" is the process of analyzing raw data and converting it into a format suitable for training a natural language processing model.

[0581] A "natural language processing model" is a machine learning model designed to understand and generate human language.

[0582] "Training" is the process of training a natural language processing model using collected data.

[0583] A "trained natural language processing model" is a natural language processing model that has completed learning through collected data.

[0584] A "user input message" is a text message that a user inputs to the system.

[0585] A "reply" is a response message generated by a natural language processing model in response to an input message.

[0586] "Display" is the act of visually presenting the generated response to the user.

[0587] "Brick-and-mortar terminals" are communication terminals such as displays and robots installed within commercial facilities.

[0588] An "application program" is software that runs on a terminal in a physical store and provides users with an interactive experience.

[0589] "Feedback" refers to the evaluation or opinion that a user gives about the system's response.

[0590] "Encryption" is the process of transforming data using a specific algorithm in order to store it securely.

[0591] This invention is a system that collects basic information and message history of the deceased, and provides an experience where users can interact with the deceased based on this information. The system is mainly composed of a server, a terminal, and a user.

[0592] Server Roles

[0593] The server performs the following main functions:

[0594] 1. Data collection and storage: The user uploads the deceased person's basic information (such as name and date of birth) and message history (LINE and text messages, etc.). The server receives this data, encrypts it, and stores it in a secure database (e.g., PostgreSQL).

[0595] 2. Model Preprocessing and Training: The collected message history is preprocessed for the natural language processing model (e.g., OpenAI's GPT-4) to be used. Preprocessing involves tokenizing the message text and removing noise. The server then uses the preprocessed data to train the natural language processing model to learn the deceased's unique speaking style and phrasing.

[0596] Device Role

[0597] The devices installed in physical stores (e.g. smart displays and robots) have the following functions:

[0598] 1. Receiving a message from the user: A message entered by the user through the terminal is sent to the server. For example, the user enters "Mom, how was your day?"

[0599] 2. Receiving and displaying a response: The generated response received from the server is displayed to the user. For example, the server generates a response such as "I may not have done much today, but I'm always thinking of you" and sends it to the terminal.

[0600] User Roles

[0601] A user uses the system as follows:

[0602] 1. Providing information about the deceased: Provide basic information and message history about the deceased when starting the system.

[0603] 2. Start of dialogue: Enter a message into a terminal in the physical store and receive a response from the server.

[0604] 3. Providing feedback: Providing feedback on the server-generated responses and contributing to the improvement of the system.

[0605] Sample prompt sentence

[0606] Based on the user's message entered on the terminal, the server generates a prompt and creates a response. For example, the following prompt is used:

[0607] User message:

[0608] "Mom, I want to cook your favorite dish. What was your favorite?"

[0609] Example prompt sentence:

[0610] Message History

[0611] User: Mom, I want to cook your favorite dish. What was your favorite?

[0612] Deceased:

[0613] Hardware and software used

[0614] Server: PostgreSQL is used as the database management system, and OpenAI's GPT-4 is used as the natural language processing model.

[0615] Terminals: Smart displays and robots are used to enable real-time interaction with users.

[0616] Secure storage: AES (Advanced Encryption Standard) encryption technology is used as the data encryption method.

[0617] This allows users to enjoy interacting with deceased loved ones even in physical stores, and the quality of the interaction experience can be improved by improving the model based on the feedback provided.

[0618] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0619] Step 1:

[0620] The user collects basic information and message history of the deceased person and uploads it to the system. Specifically, the user sends text files and LINE message history to the server via their device. The input is the deceased person's information and message history, and the output is the transmission of this data to the server.

[0621] Step 2:

[0622] The server receives the collected basic information and message history of the deceased and stores it in a secure database. Specifically, after receiving the data, it encrypts it using AES encryption technology and stores it in a PostgreSQL database. The input is the data sent by the user, and the output is stored in the database in encrypted form.

[0623] Step 3:

[0624] The server preprocesses the stored message history by tokenizing the message text and removing unnecessary noise (e.g., emojis and special symbols). The input is the message history retrieved from the database, and the output is the tokenized and preprocessed text data.

[0625] Step 4:

[0626] The server uses the preprocessed data to train a natural language processing model. Specifically, it uses the preprocessed dataset to train OpenAI's GPT-4 model to learn the deceased's unique speaking style and phrasing. The input is the preprocessed text data, and the output is the trained natural language processing model.

[0627] Step 5:

[0628] The user initiates a conversation with the deceased through a terminal in a physical store. Specifically, the user enters a message into the terminal and presses the "send" button. The input is the message the user enters into the terminal, and the output is the message being sent to the server.

[0629] Step 6:

[0630] The terminal sends the input message from the user to the server. Specifically, it adds metadata such as a timestamp and user ID to the input message and sends it to the server via the API. The input is the user's input message and metadata, and the output is that the message is sent to the server.

[0631] Step 7:

[0632] The server generates a response using the input message received from the user, specifically by generating a prompt sentence like the following and feeding it to the GPT-4 model:

[0633] Message History

[0634] User: Mom, I want to cook your favorite dish. What was your favorite?

[0635] Deceased:

[0636] The input is the user's input message and prompt, and the output is the generated response.

[0637] Step 8:

[0638] The server returns the generated response to the terminal. Specifically, it sends a response message to the terminal through the API. The input is the generated response, and the output is the response message returned to the terminal.

[0639] Step 9:

[0640] The terminal displays the response received from the server to the user. Specifically, a smart display or robot conveys the response message to the user visually or audibly. The input is the response message sent from the server, and the output is the response displayed to the user.

[0641] Step 10:

[0642] The user provides feedback on the system-generated response by pressing a button to rate whether the response was appropriate. The input is the user's rating, and the output is that rating is sent to the server.

[0643] Step 11:

[0644] The server collects user feedback and uses it to improve the natural language processing model. Specifically, it analyzes the feedback information and uses it to retrain the model. The input is the user feedback, and the output is an improved natural language processing model.

[0645] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0646] The present invention is a system that provides a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing the user with responses generated by a natural language processing model and an emotion engine. The program processing of the system is explained below step by step.

[0647] Overall system configuration

[0648] This system consists of a server, a terminal, and a user. The role of each component is explained in detail below.

[0649] 1. Data Collection

[0650] To start using the system, users enter and upload basic information about the deceased person and their LINE message history. For example, imagine a situation where a user enters the name and date of birth of the deceased person and uploads a text file of past LINE messages.

[0651] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server in a secure manner.

[0652] 2. Data storage

[0653] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[0654] 3. Data Learning

[0655] The server preprocesses the stored message history and trains a natural language processing model, which includes tokenizing the message text, removing unnecessary data, and morphological analysis. The preprocessed data is used to train the model on the deceased person's unique speaking and phrasing patterns.

[0656] 4. Generating dialogue with the user

[0657] A user uses a terminal to enter a message to initiate a conversation with the system. For example, a user might enter, "How was your day?"

[0658] The terminal sends the input message from the user to the server, with necessary metadata (such as a timestamp and user ID) added to the input message.

[0659] The server analyzes the message received from the user and generates an appropriate response using a trained natural language processing model. The emotion engine is also utilized here. Specifically, it recognizes the emotion in the user's input message and adjusts the tone and content of the response based on that emotion. For example, if the user says, "Today was a tough day," the emotion engine recognizes the "tough" emotion and adjusts the response to be more warmhearted.

[0660] The device displays the response received from the server to the user, allowing the user to experience the experience of interacting with the deceased.

[0661] 5. Ongoing dialogue and feedback

[0662] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[0663] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion engine. It analyzes the feedback data and retrains it to improve the quality of responses.

[0664] 6. Data Security

[0665] The server uses encryption methods for storing and transmitting collected data, which ensures the protection of user privacy.

[0666] Through these steps, users can ease their grief through an advanced dialogue experience. Furthermore, by utilizing an emotion engine, the system can generate responses that are in tune with the user's emotions, enabling more personalized care.

[0667] The processing flow will be explained below.

[0668] Step 1:

[0669] The user uses the device to enter basic information about the deceased person and their past message history and upload it. For example, the user enters the deceased person's name and date of birth, selects the LINE message history as a text file, and presses the send button.

[0670] Step 2:

[0671] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server using a secure communication protocol.

[0672] Step 3:

[0673] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[0674] Step 4:

[0675] The server preprocesses the stored message history, which includes tokenizing the message text and removing unnecessary data, and converts the formatted data into a format suitable for training natural language processing models.

[0676] Step 5:

[0677] The server uses the preprocessed data to train a natural language processing model. During the training process, the model learns the deceased person's unique speech and phrasing patterns. Through this process, the model learns the deceased person's linguistic style.

[0678] Step 6:

[0679] A user inputs a message through a terminal to start a conversation with the system, for example, "How was your day?", and presses the send button.

[0680] Step 7:

[0681] The device sends the input message to the server. The message is accompanied by necessary metadata (such as a timestamp and user ID) and is forwarded to the server while ensuring reliability.

[0682] Step 8:

[0683] The server analyzes messages received from users and uses an emotion engine to recognize the user's emotions. For example, if a user types "I had a hard time today," the emotion engine will recognize "sadness" or "stress" from the user's message.

[0684] Step 9:

[0685] The server uses a natural language processing model to generate an appropriate response based on the emotional data recognized by the emotion engine. The response is tailored to reflect the user's emotions. For example, in response to an input such as "Today was tough," a warm response such as "That must have been really tough. Thank you for talking to me" is generated.

[0686] Step 10:

[0687] The server then sends the generated response to the terminal, converting the response message into the required format so that it can be properly displayed to the user.

[0688] Step 11:

[0689] The device displays the responses received from the server to the user, allowing the user to experience the feeling of having a conversation with the deceased. Furthermore, the responses are emotionally sensitive, providing a deeper interaction experience.

[0690] Step 12:

[0691] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[0692] Step 13:

[0693] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion engine. The feedback data is analyzed and retrained to improve the quality of responses. This effort allows for continuous improvement of the overall system performance and user satisfaction.

[0694] This allows users to ease their grief through a sophisticated dialogue experience, allowing the system to provide responses that are sensitive to the user's emotions, while also ensuring data security and privacy.

[0695] Example 2

[0696] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0697] In modern society, people miss their deceased loved ones, but there are limited means to recreate their memories and messages. However, existing technologies have not yet fully developed systems that provide a dialogue experience with the deceased, making it difficult for users to experience the sensation of "talking" with the deceased. Therefore, new methods are needed to provide users with psychological comfort and healing by maintaining contact with the deceased.

[0698] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0699] In this invention, the server includes means for a user to input and upload basic information about the deceased and their communication history, means for securely processing the input and uploaded data and transferring it to the server, means for decompressing and decrypting the received data and storing it in a database, means for preprocessing the stored communication history, tokenizing and analyzing the data, and training a natural language processing model, means for receiving an input message from the user and generating a response using the trained natural language processing model and an emotion estimation engine, and means for displaying the generated response to the user, thereby enabling the user to have a conversation experience with the deceased.

[0700] "Deceased" refers to someone who has already died.

[0701] "Basic information" refers to the main data that can be used to identify an individual, such as name and date of birth.

[0702] "Communication history" refers to a record of past messages and conversations.

[0703] "User" refers to an individual who uses this system.

[0704] "Input and upload" refers to the act of a user providing data to the system.

[0705] "Secure processing" refers to taking measures such as compression and encryption to handle data safely.

[0706] "Server" refers to a computer system that stores and processes data received from users.

[0707] "Decompressing and decrypting" refers to restoring compressed data to its original form and decrypting encrypted data.

[0708] A "database" refers to a system for organizing and storing data.

[0709] "Preprocessing" refers to processes such as data cleaning and tokenization to make the data suitable for training a model.

[0710] "Tokenization" refers to the process of dividing text data into meaningful units.

[0711] A "natural language processing model" refers to a machine learning algorithm for understanding and generating human language.

[0712] An "emotion estimation engine" refers to an algorithm that analyzes and estimates emotions from input text data.

[0713] "Generating a reply" refers to creating an appropriate response to an input message.

[0714] "Display" refers to outputting the generated response to the user's terminal screen.

[0715] This invention is a system that provides a conversational experience with a deceased person. It collects basic information and communication history of the deceased person and provides the user with responses generated by a natural language processing model and an emotion estimation engine. The system is composed of a server, a terminal, and a user.

[0716] Data Collection and Transmission

[0717] To use the system, users must first enter and upload basic information and communication history of the deceased. Specifically, users enter the name and date of birth of the deceased and upload past communication history (e.g., text file format of messages).

[0718] The terminal receives this data, compresses and encrypts it, and transfers it to the server. For data compression, it uses the zip library, and for encryption, it uses the AES encryption library. The data is sent to the server using a secure communication protocol (for example, HTTPS).

[0719] Receiving and storing data

[0720] The server receives the encrypted data, unpacks and decrypts it using a zip library for unpacking and an AES decryption library for decryption. The server then stores the unpacked and decrypted data in a database (e.g., MySQL).

[0721] Data Learning

[0722] The server preprocesses the stored communication history and trains a natural language processing model, which includes tokenization and removal of unnecessary data using morphological analysis tools (e.g., MeCab), to learn the deceased person's unique speech patterns and phrasing.

[0723] User interaction

[0724] A user uses a terminal to enter a message to initiate a conversation with the system. For example, a user might enter, "How was your day?"

[0725] The terminal sends the input message from the user to the server, with metadata such as a timestamp and user ID attached to the message.

[0726] The server analyzes messages received from users and generates appropriate responses using a trained natural language processing model and an emotion estimation engine. For example, if a user says, "Today was a tough day," the emotion engine recognizes the "toughness" and adjusts the response to be more warm-hearted.

[0727] The device displays the response received from the server to the user, allowing the user to experience the experience of interacting with the deceased.

[0728] Continuous feedback and improvement

[0729] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[0730] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion estimation engine. It analyzes the feedback data and retrains it to improve the quality of responses.

[0731] Data Security

[0732] The server uses encryption methods when storing and transmitting collected data, thereby ensuring user privacy.

[0733] Examples of concrete examples and prompts

[0734] For example, if a user sends a message saying "How was your day?", the server will do the following:

[0735] 1. The user uses a terminal to type, "How was your day?"

[0736] 2. The device sends this message to the server.

[0737] 3. The server analyzes the message and uses an emotion engine to recognize the user's emotion.

[0738] 4. The trained natural language processing model generates a response, for example, "I was thinking of you today, how are you?"

[0739] 5. The server generates a response and sends it to the terminal.

[0740] 6. The terminal displays the reply to the user.

[0741] In this way, users can enjoy the interactive experience of interacting with the deceased, and continuous feedback improves the accuracy of the system.

[0742] Examples of prompts include the following:

[0743] User: How was your day?

[0744] Model: I was thinking of you today, how are you?

[0745] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0746] Step 1:

[0747] The user enters and uploads basic information and communication history of the deceased.

[0748] Input: Basic information of the deceased (e.g. name, date of birth), text file of communication history.

[0749] Specific operation: The user enters the name of the deceased in the name input field, the date of birth of the deceased in the date of birth field, selects the text file of the communication history using the file selection function, and presses the upload button.

[0750] Output: User-entered data sent to the terminal.

[0751] Step 2:

[0752] The terminal receives data entered and uploaded by the user, compresses and encrypts it, and transfers it to the server.

[0753] Input: User-entered data (basic information of the deceased, text file of communication history)

[0754] Specific operation: The terminal compresses the data using a data compression library (e.g., zip library), encrypts the compressed data using an encryption library (e.g., AES encryption), and transfers it to the server using the HTTPS protocol.

[0755] Output: The encrypted and compressed data sent to the server.

[0756] Step 3:

[0757] The server receives the encrypted data, decompresses and decrypts it, and stores it in a database.

[0758] Input: Encrypted and compressed data.

[0759] What happens: The server decrypts the data using an AES decryption library, unpacks it using a zip library, and then stores the unpacked data appropriately in the database system (e.g., MySQL).

[0760] Output: The decompressed and decrypted data is stored in the database.

[0761] Step 4:

[0762] The server preprocesses the stored communication history and trains a natural language processing model.

[0763] Input: The deceased person's communication history stored in a database.

[0764] What it does: The server uses a natural language processing library (e.g., NLTK, spaCy) to tokenize the message text and remove unnecessary data. The preprocessed data is used to train a model to learn the unique speaking style and phrasing of the deceased.

[0765] Output: A trained natural language processing model.

[0766] Step 5:

[0767] The user uses the terminal to enter messages to interact with the system.

[0768] Input: A message typed by the user (e.g., "How was your day?")

[0769] Specific operation: The user enters a message in the input form and presses the send button.

[0770] Output: User input messages sent to the terminal.

[0771] Step 6:

[0772] The terminal sends an input message from the user to the server.

[0773] Input: User input message.

[0774] Specific operation: The terminal adds metadata such as a timestamp and user ID to the input message and sends it to the server using the HTTPS protocol.

[0775] Output: The user's input message and metadata sent to the server.

[0776] Step 7:

[0777] The server analyzes messages received from users and generates appropriate responses using a trained natural language processing model and emotion estimation engine.

[0778] Input: The message and metadata sent by the user.

[0779] How it works: The server analyzes the message content using a natural language processing library and identifies the user's sentiment using a sentiment analysis library (e.g., TextBlob, VADER). A trained generative AI model generates a response, and the sentiment estimation engine adjusts the response.

[0780] Output: The generated reply message.

[0781] Step 8:

[0782] The terminal displays the response received from the server to the user.

[0783] Input: The reply message sent by the server.

[0784] Specific operation: The terminal displays the received reply message on the screen, visually presenting it to the user.

[0785] Output: The reply message that is displayed to the user.

[0786] Step 9:

[0787] The user provides feedback on the system's response.

[0788] Input: User ratings and comments.

[0789] Specific operation: The user enters the response rating and comments in the feedback form and presses the submit button.

[0790] Output: Feedback data sent to the device.

[0791] Step 10:

[0792] The server collects feedback from users and reflects it in improving the natural language processing model and emotion estimation engine.

[0793] Input: Feedback data from users.

[0794] What it does: The server stores the feedback data in a database, analyzes it to identify areas for improvement, and retrains the model.

[0795] Output: Improved natural language processing models and sentiment estimation engines.

[0796] (Application example 2)

[0797] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0798] Conventional dialogue systems have difficulty providing a dialogue experience with the deceased, and lack emotional care when users interact with the deceased through past message history. Furthermore, there is no function to provide support for products related to the deceased within the virtual store, making it difficult for users to select products that are considerate to the deceased. Our goal is to solve these issues and provide a more personal and emotionally sensitive dialogue experience.

[0799] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0800] In this invention, the server includes means for collecting and storing basic information and communication history of the deceased, means for preprocessing the collected communication history to train a natural language processing model, means for receiving an input message from a user and generating a response using the trained natural language processing model, means for displaying the generated response to the user, means for providing support for inquiries about products related to the deceased in a virtual store, and means for adjusting the tone and content of the response using an emotion engine. This enables the user to purchase products related to the deceased in the virtual store while receiving personal support through a dialogue with the deceased.

[0801] "Deceased" refers to a person who has passed away.

[0802] "Basic information" refers to personal data such as the deceased's name and date of birth.

[0803] "Communication history" refers to past messages and conversation records between the deceased and the user.

[0804] A "natural language processing model" is an algorithm that understands and processes human language, generating responses based on input text.

[0805] "Preprocessing" refers to the process of preparing collected data in a form that is easy to analyze by tokenizing it, removing unnecessary data, performing morphological analysis, etc.

[0806] "User" refers to a person who uses this system to have an interactive experience with a deceased person.

[0807] "Input message" refers to a text message that a user sends to the system.

[0808] "Reply" refers to a response message generated by the system that mimics the speech style and emotions of the deceased.

[0809] "Display means" refers to a method or device for displaying the generated response on the user's terminal.

[0810] A "virtual store" refers to a virtual store space where products and services can be purchased online.

[0811] An "emotion engine" is an algorithm that analyzes emotions from a user's input message and adjusts the tone and content of the response based on those emotions.

[0812] The present invention is a system that allows users to have a conversational experience with a deceased person. It collects and stores basic information and communication history of the deceased, and can train a natural language processing model based on that data. Furthermore, it has the function of providing support for inquiries about products related to the deceased in a virtual store.

[0813] Overall system configuration

[0814] This system is composed of a server, a terminal, and a user, and operates as follows.

[0815] Data collection and storage

[0816] To start using the system, users input and upload basic information about the deceased person and their communication history, such as their name, date of birth, and past message history in text format. The device receives this information, compresses and encrypts the data, and sends it to the server.

[0817] The server decompresses and decrypts the received data and stores it in a database. When storing it, it properly associates the deceased person's basic information and message history with the data and stores it in secure storage, thereby protecting the user's privacy.

[0818] Data preprocessing and model training

[0819] The server preprocesses the stored message history, which includes tokenizing the messages, removing unnecessary data, and morphological analysis. The preprocessed data is then used to train a natural language processing model to learn the deceased person's unique phrasing and speaking patterns.

[0820] User interaction generation and display

[0821] The user uses the device to input a message to initiate a conversation with the system. For example, they could ask, "Did the deceased have any special memories of this accessory?" The device then sends this input message to the server, which then analyzes the message and generates an appropriate response using a trained natural language processing model. The server also uses an emotion engine to recognize emotions in the user's input message and adjust the tone and content of the response based on those emotions.

[0822] The device displays the generated response to the user, allowing the user to experience as if they were having a conversation with the deceased person.

[0823] Support functions within the virtual store

[0824] The system includes a function that provides support for inquiries about products related to the deceased in the virtual store. For example, when a user finds an accessory related to the deceased in the virtual store, they can ask about the deceased's memories and stories associated with the accessory and receive appropriate answers. This allows users to select products through a more personalized experience.

[0825] Software and hardware used

[0826] Server: Used for storing data, preprocessing, training natural language processing models, and generating responses.

[0827] Device: The device (smartphone, tablet, etc.) that a user uses to type and receive messages.

[0828] Natural language processing model: Using OpenAI GPT-3 and other models, responses are generated based on the user's input message.

[0829] Emotion engine: Analyzes the sentiment of a user's message and adjusts the tone and content of responses accordingly.

[0830] Examples of concrete examples and prompts

[0831] For example, if a user is searching for an accessory related to a deceased person in a virtual store, a possible question might be, "Did the deceased have any special memories associated with this accessory?"

[0832] Example prompt sentence:

[0833] How was your day?

[0834] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0835] Step 1:

[0836] Data collection

[0837] Users input and upload basic information and communication history of the deceased person into the device. Specifically, they provide the deceased person's name, date of birth, and past message history as a text file. The device receives this data, compresses and encrypts it, and sends it to the server.

[0838] Input: Deceased person's name, date of birth, message history.

[0839] Output: Compressed and encrypted data.

[0840] Step 2:

[0841] Data storage

[0842] The server receives the data sent from the device, decompresses and decrypts it, and then associates the deceased person's basic information and message history with a database and stores them in a secure storage environment to ensure the data remains private.

[0843] Input: Compressed and encrypted data.

[0844] Output: Basic information and message history of the deceased person stored in the database.

[0845] Step 3:

[0846] Data Preprocessing

[0847] The server preprocesses the stored message history by tokenizing the messages, removing unnecessary data, and performing morphological analysis. This process prepares the data in a format that is easy to analyze.

[0848] Input: The saved message history.

[0849] Output: Preprocessed data.

[0850] Step 4:

[0851] Training the model

[0852] The server uses the preprocessed data to train a natural language processing model, which learns the specific phrasing and speech patterns of the deceased.

[0853] Input: Preprocessed data.

[0854] Output: A trained natural language processing model.

[0855] Step 5:

[0856] Dialogue Generation

[0857] The user inputs a message to the system using the terminal, for example, a question such as "Did the deceased have any special memories about this accessory?" The terminal then sends this input message to the server.

[0858] Input: The user's input message.

[0859] Output: The input message sent to the server.

[0860] Step 6:

[0861] Response Generation

[0862] The server analyzes messages sent by users and generates appropriate responses using a trained natural language processing model, including an emotion engine to analyze the user's emotions and adjust the tone and content of the response.

[0863] Input: The input message sent to the server.

[0864] Output: The generated response.

[0865] Step 7:

[0866] Viewing the response

[0867] The terminal displays the generated response received from the server to the user, thereby allowing the user to have a conversational experience with the deceased person.

[0868] Input: The generated response.

[0869] Output: The response displayed to the user.

[0870] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0871] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0872] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0873] [Third embodiment]

[0874] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0875] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0876] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0877] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0878] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0879] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0880] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0881] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0882] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0883] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0884] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0885] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0886] The present invention is a system that provides a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing the user with responses generated by a natural language processing model. The program processing of the system is explained below in order.

[0887] Overall system configuration

[0888] This system mainly consists of a server, a terminal, and a user. The roles of each component are as follows:

[0889] 1. Data Collection

[0890] To start using the system, the user provides basic information about the deceased person (such as name and date of birth) and their LINE message history. As a concrete example, we will assume a situation where the user uploads the deceased person's past LINE messages to their device as a text file.

[0891] The server receives the uploaded data and stores it in a secure database, converts it into a suitable format, and stores it encrypted to protect privacy.

[0892] 2. Data Learning

[0893] The server preprocesses the stored message history to train a natural language processing model. Preprocessing includes, for example, tokenizing the message text and removing unnecessary noise. For example, the server analyzes the grammar and syntax of LINE messages to create a dataset for machine learning.

[0894] The server uses natural language processing techniques to train the model to recognize the deceased person's unique speaking style and expressions. During the training process, patterns are extracted from message history and reflected in the model. For example, the model learns the deceased person's frequently used phrases and unique expressions.

[0895] 3. Generating dialogue with the user

[0896] The user enters a message at the terminal to initiate a dialogue with the system. The user can enter, for example, "How was your day?"

[0897] The terminal sends input from the user to the server. At this time, the terminal adds necessary metadata (e.g., a timestamp) to the input message and sends it to the server. As a concrete example, the terminal sends the user's message "How was your day?" to the server.

[0898] The server generates a response based on the message received from the user, using a trained natural language processing model to generate a response that the deceased person would likely respond with, for example, "I may not have done much today, but I'm always thinking of you."

[0899] The terminal displays the response received from the server to the user, allowing the user to have an experience that feels as if they are interacting with the deceased.

[0900] 4. Ongoing dialogue and feedback

[0901] The user can provide feedback on the system's response, for example by being presented with a button to rate whether the response was appropriate.

[0902] The server improves the natural language processing model based on user feedback. The collected feedback information is used to further train the model. For example, the accuracy of the model can be improved by relearning response patterns where users have given positive feedback.

[0903] This allows users to ease their grief through conversations with the deceased. Data privacy is also ensured, making this a system that users can use with peace of mind.

[0904] The processing flow will be explained below.

[0905] Step 1:

[0906] The user uses the device to enter and upload the deceased person's basic information and LINE message history. Specifically, the user enters the deceased person's name and date of birth into the form, selects a text file of past LINE messages, and presses the send button.

[0907] Step 2:

[0908] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server in a secure manner.

[0909] Step 3:

[0910] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[0911] Step 4:

[0912] The server preprocesses the stored message history, including tokenizing the message text, removing unnecessary data, and performing morphological analysis, so that the formatted data is suitable for training natural language processing models.

[0913] Step 5:

[0914] The server uses the preprocessed data to train a natural language processing model. During the training process, the model learns the deceased person's unique speech patterns and phrasing. Specifically, a recurrent neural network and a transformer model are used to build a language model of the deceased person.

[0915] Step 6:

[0916] A user uses a terminal to enter a message to initiate a conversation with the system, for example, "How was your day?", into an input field on the terminal and presses the send button.

[0917] Step 7:

[0918] The terminal receives an input message from the user and sends it to the server, adding necessary metadata (such as a timestamp or user ID) to the input message and ensuring its reliability.

[0919] Step 8:

[0920] The server analyzes messages received from users and uses a trained natural language processing model to generate appropriate responses, which are naturally crafted based on the linguistic style of the deceased.

[0921] Step 9:

[0922] The server then sends the generated response to the terminal, converting the response message into the required format so that it can be properly displayed to the user.

[0923] Step 10:

[0924] The terminal displays the response received from the server to the user, who can then experience the experience of interacting with the deceased person.

[0925] Step 11:

[0926] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[0927] Step 12:

[0928] The server stores the feedback collected from users and reflects it in improving the natural language processing model, analyzing the feedback data and retraining it to improve the quality of responses.

[0929] These steps allow users to ease their grief through an enhanced interactive experience, while ensuring data security and privacy.

[0930] Example 1

[0931] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0932] Currently, there are systems that provide users who want to reminisce about their deceased loved ones with an experience that makes it seem as if they are having a conversation with the deceased. However, these systems do not adequately address issues such as the security of the information provided by the user, continuous improvement of the model based on user feedback, and learning and reproducing the unique speaking style and expressions of the deceased. This makes it difficult for users to obtain a natural and realistic conversational experience.

[0933] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0934] In this invention, the server includes means for collecting and storing basic information and message history of the deceased, means for preprocessing the collected message history to train a natural language processing model, means for receiving an input message from a user and generating a reply using the trained natural language processing model, means for displaying the generated reply to the user, and means for adding metadata such as a timestamp to the input message from the user and transmitting it to the server. This makes it possible to securely store data, improve the model using feedback, and reproduce the deceased's unique speaking style and expressions, thereby providing the user with a more natural and realistic conversation experience.

[0935] "Basic information about the deceased" refers to the name, date of birth, and other basic information needed to identify the deceased.

[0936] "Message history" refers to records of text messages and chats exchanged by a deceased person during their lifetime, and is primarily used as training data for natural language processing models.

[0937] A "natural language processing model" is a machine learning algorithm designed to understand and generate human language, generating appropriate responses to messages input by users.

[0938] "Tokenization" is a preprocessing technique that divides text data into smaller units such as words or phrases.

[0939] "Preprocessing" refers to a series of processes that remove unnecessary noise from collected data and prepare it in a format suitable for training a natural language processing model.

[0940] A "timestamp" is information indicating the date and time when a specific event occurred, and is added to a message input by a user.

[0941] "Feedback" refers to the user's evaluation or opinion on the system's response, and is data used to improve the model.

[0942] A "trained natural language processing model" is a model that has learned based on the deceased person's message history and is capable of reproducing the deceased person's unique speaking style and phrasing.

[0943] "Encryption" is the process of converting collected data using a specific algorithm so that it cannot be easily analyzed or viewed by third parties.

[0944] A "database" is a system for systematically storing and managing collected data.

[0945] The present invention provides a system for providing a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing a user with responses generated by a natural language processing model. Specific embodiments of the system are described below.

[0946] The hardware required to implement this system is a server, user terminals, and a network connection. The software includes a database management system for data collection and storage, a natural language processing model, and encryption algorithms.

[0947] Data collection and storage

[0948] To begin using the system, users first provide basic information about the deceased (such as name and date of birth) and their message history. This information is primarily in the form of a text file, and users upload it to the system using their device. Specifically, users prepare LINE messages with the deceased as a text file and upload it to their device.

[0949] The device collects the information uploaded by the user and sends it to the server, which stores the received information in a database. At this time, the data is converted into an appropriate format, encrypted, and stored securely.

[0950] Data Preprocessing and Training

[0951] The server pre-processes the stored message history, which includes tokenizing the messages and removing unnecessary noise. For example, the server removes special characters and unnecessary whitespace from the message history and tokenizes it into words.

[0952] The pre-processed data is used to train a natural language processing model to learn the distinctive speech patterns and phrasing of the deceased, which the server uses to build the model and extract patterns and features from the messages.

[0953] User interaction generation

[0954] A user types a message at a terminal to initiate a dialogue with the system. For example, the user types "How was your day?"

[0955] The device sends the message entered by the user, along with metadata such as a timestamp, to the server. The server then analyzes the message and uses a trained natural language processing model to generate a response, such as, "I may not have done much today, but I'm always thinking of you."

[0956] The device displays the generated response to the user, allowing the user to experience as if they were interacting with the deceased.

[0957] Feedback and Improvements

[0958] The user provides feedback on the system's response by providing a button to rate whether the response is appropriate or not.

[0959] The server improves the natural language processing model based on user feedback. By collecting feedback data and retraining the model based on it, it becomes possible to generate more accurate responses.

[0960] Specific examples

[0961] For example, if a user wishes to reminisce about a deceased loved one, they could write a prompt like this:

[0962] "What was your favorite movie?"

[0963] "Tell me about the last trip you took together."

[0964] As the user enters these prompts, the system infers how the deceased would have answered these questions and generates an appropriate response, such as, "My favorite movie was 'Favorite Movie Title.' I always enjoyed watching it with you."

[0965] Through this system, users can enjoy the experience of interacting with the deceased and ease the grief of bereavement.In addition, data privacy is ensured, so users can use the system with peace of mind.

[0966] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0967] Step 1:

[0968] Data collection

[0969] To start using the system, the user uploads the deceased's basic information (such as name and date of birth) and the message history with the deceased to the device. Specifically, the user provides the LINE messages with the deceased in the form of a text file to the device. The input is the deceased's basic information and message history text file, and the output is the input data saved on the device.

[0970] The terminal reads the uploaded text file. Specifically, the terminal loads the file contents into memory, checks the data format, and then sends the data to the server.

[0971] Step 2:

[0972] Data storage

[0973] The server securely receives the data sent from the device and stores it in a database. At this time, the data is converted into an easy-to-analyze format and encrypted. The input is the basic information and message history of the deceased person sent from the device, and the output is securely stored in the database.

[0974] Specifically, the server stores basic information about the deceased in specific fields, converts message history from plain text to JSON format, and encrypts the data using an encryption algorithm to protect privacy.

[0975] Step 3:

[0976] Data Preprocessing

[0977] The server preprocesses the stored message history, which includes tokenization and removal of unnecessary noise. The input is the message history data retrieved from the database, and the output is the preprocessed dataset.

[0978] Specifically, the server removes special characters and unnecessary spaces from the message history, processes it into a grammatically separable form, tokenizes the data, and splits it into words or phrases. Furthermore, during the noise removal process, it removes things like stop verbs, which are generally considered meaningless.

[0979] Step 4:

[0980] Training a natural language processing model

[0981] The server uses the preprocessed dataset to train a natural language processing model, where the input is the preprocessed dataset and the output is the trained natural language processing model.

[0982] Specifically, the server runs machine learning algorithms to extract patterns and features from the message history, training the model to recognize the unique speech patterns and phrasing of the deceased, fine-tuning specific phrases and unique expressions.

[0983] Step 5:

[0984] User interaction generation

[0985] A user enters a message at a terminal to initiate a conversation with the system, for example, entering a prompt such as "How was your day?" The input is the message from the user, and the output is the message sent to the server.

[0986] The terminal sends the message entered by the user to the server, with metadata such as a timestamp attached to the message. Specifically, the terminal formats the message, adds a timestamp, and sends it to the server over the network.

[0987] The server analyzes the received message and generates a response using a trained natural language processing model. The input is the user's message, and the output is the generated response. For example, the server might generate a response like, "I may not have done much today, but I'm always thinking of you."

[0988] The device receives the generated response and displays it to the user, allowing the user to experience the experience as if they were interacting with the deceased person.

[0989] Step 6:

[0990] Feedback and model improvement

[0991] The user provides feedback on the system's response, for example by clicking a button to rate whether the response was appropriate. The input is the feedback from the user, and the output is the feedback data to the server.

[0992] The terminal transmits the user's feedback to the server, specifically, the terminal collects the feedback information and transmits it together with the metadata to the server.

[0993] The server improves the natural language processing model based on user feedback. The input is the feedback data, and the output is an improved natural language processing model. Specifically, the server uses the feedback to retrain the model and improve the accuracy of responses.

[0994] (Application example 1)

[0995] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0996] Many people seek peace of mind through conversations with their deceased loved ones. However, current technology makes it difficult to provide such experiences in real time. Furthermore, systems that provide such conversational experiences are typically limited to home or personal devices, and are rarely available in commercial facilities or brick-and-mortar stores. This presents a challenge in providing valuable experiences for customers, contributing to attracting more customers and improving customer satisfaction.

[0997] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0998] In this invention, the server includes means for collecting and storing basic information and message history of the deceased, means for preprocessing the collected message history to train a natural language processing model, and means for receiving input messages from users and generating responses using the trained natural language processing model. This allows users to have the experience of interacting with the deceased through an application program installed on a terminal in a physical store. This allows users to enjoy a special conversation experience based on their memories of the deceased in real time at commercial facilities or physical stores, which is expected to increase customer attraction and customer satisfaction.

[0999] "Basic information about the deceased" refers to basic personal information about the deceased, such as name, date of birth, and gender.

[1000] A "message history" is a record of text messages exchanged between the user and the deceased person.

[1001] "Collection" is the process of obtaining the required data from the user and storing that data.

[1002] "Preprocessing" is the process of analyzing raw data and converting it into a format suitable for training a natural language processing model.

[1003] A "natural language processing model" is a machine learning model designed to understand and generate human language.

[1004] "Training" is the process of training a natural language processing model using collected data.

[1005] A "trained natural language processing model" is a natural language processing model that has completed learning through collected data.

[1006] A "user input message" is a text message that a user inputs to the system.

[1007] A "reply" is a response message generated by a natural language processing model in response to an input message.

[1008] "Display" is the act of visually presenting the generated response to the user.

[1009] "Brick-and-mortar terminals" are communication terminals such as displays and robots installed within commercial facilities.

[1010] An "application program" is software that runs on a terminal in a physical store and provides users with an interactive experience.

[1011] "Feedback" refers to the evaluation or opinion that a user gives about the system's response.

[1012] "Encryption" is the process of transforming data using a specific algorithm in order to store it securely.

[1013] This invention is a system that collects basic information and message history of the deceased, and provides an experience where users can interact with the deceased based on this information. The system is mainly composed of a server, a terminal, and a user.

[1014] Server Roles

[1015] The server performs the following main functions:

[1016] 1. Data collection and storage: The user uploads the deceased person's basic information (such as name and date of birth) and message history (LINE and text messages, etc.). The server receives this data, encrypts it, and stores it in a secure database (e.g., PostgreSQL).

[1017] 2. Model Preprocessing and Training: The collected message history is preprocessed for the natural language processing model (e.g., OpenAI's GPT-4) to be used. Preprocessing involves tokenizing the message text and removing noise. The server then uses the preprocessed data to train the natural language processing model to learn the deceased's unique speaking style and phrasing.

[1018] Device Role

[1019] The devices installed in physical stores (e.g. smart displays and robots) have the following functions:

[1020] 1. Receiving a message from the user: A message entered by the user through the terminal is sent to the server. For example, the user enters "Mom, how was your day?"

[1021] 2. Receiving and displaying a response: The generated response received from the server is displayed to the user. For example, the server generates a response such as "I may not have done much today, but I'm always thinking of you" and sends it to the terminal.

[1022] User Roles

[1023] A user uses the system as follows:

[1024] 1. Providing information about the deceased: Provide basic information and message history about the deceased when starting the system.

[1025] 2. Start of dialogue: Enter a message into a terminal in the physical store and receive a response from the server.

[1026] 3. Providing feedback: Providing feedback on the server-generated responses and contributing to the improvement of the system.

[1027] Sample prompt sentence

[1028] Based on the user's message entered on the terminal, the server generates a prompt and creates a response. For example, the following prompt is used:

[1029] User message:

[1030] "Mom, I want to cook your favorite dish. What was your favorite?"

[1031] Example prompt sentence:

[1032] Message History

[1033] User: Mom, I want to cook your favorite dish. What was your favorite?

[1034] Deceased:

[1035] Hardware and software used

[1036] Server: PostgreSQL is used as the database management system, and OpenAI's GPT-4 is used as the natural language processing model.

[1037] Terminals: Smart displays and robots are used to enable real-time interaction with users.

[1038] Secure storage: AES (Advanced Encryption Standard) encryption technology is used as the data encryption method.

[1039] This allows users to enjoy interacting with deceased loved ones even in physical stores, and the quality of the interaction experience can be improved by improving the model based on the feedback provided.

[1040] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1041] Step 1:

[1042] The user collects basic information and message history of the deceased person and uploads it to the system. Specifically, the user sends text files and LINE message history to the server via their device. The input is the deceased person's information and message history, and the output is the transmission of this data to the server.

[1043] Step 2:

[1044] The server receives the collected basic information and message history of the deceased and stores it in a secure database. Specifically, after receiving the data, it encrypts it using AES encryption technology and stores it in a PostgreSQL database. The input is the data sent by the user, and the output is stored in the database in encrypted form.

[1045] Step 3:

[1046] The server preprocesses the stored message history by tokenizing the message text and removing unnecessary noise (e.g., emojis and special symbols). The input is the message history retrieved from the database, and the output is the tokenized and preprocessed text data.

[1047] Step 4:

[1048] The server uses the preprocessed data to train a natural language processing model. Specifically, it uses the preprocessed dataset to train OpenAI's GPT-4 model to learn the deceased's unique speaking style and phrasing. The input is the preprocessed text data, and the output is the trained natural language processing model.

[1049] Step 5:

[1050] The user initiates a conversation with the deceased through a terminal in a physical store. Specifically, the user enters a message into the terminal and presses the "send" button. The input is the message the user enters into the terminal, and the output is the message being sent to the server.

[1051] Step 6:

[1052] The terminal sends the input message from the user to the server. Specifically, it adds metadata such as a timestamp and user ID to the input message and sends it to the server via the API. The input is the user's input message and metadata, and the output is that the message is sent to the server.

[1053] Step 7:

[1054] The server generates a response using the input message received from the user, specifically by generating a prompt sentence like the following and feeding it to the GPT-4 model:

[1055] Message History

[1056] User: Mom, I want to cook your favorite dish. What was your favorite?

[1057] Deceased:

[1058] The input is the user's input message and prompt, and the output is the generated response.

[1059] Step 8:

[1060] The server returns the generated response to the terminal. Specifically, it sends a response message to the terminal through the API. The input is the generated response, and the output is the response message returned to the terminal.

[1061] Step 9:

[1062] The terminal displays the response received from the server to the user. Specifically, a smart display or robot conveys the response message to the user visually or audibly. The input is the response message sent from the server, and the output is the response displayed to the user.

[1063] Step 10:

[1064] The user provides feedback on the system-generated response by pressing a button to rate whether the response was appropriate. The input is the user's rating, and the output is that rating is sent to the server.

[1065] Step 11:

[1066] The server collects user feedback and uses it to improve the natural language processing model. Specifically, it analyzes the feedback information and uses it to retrain the model. The input is the user feedback, and the output is an improved natural language processing model.

[1067] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1068] The present invention is a system that provides a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing the user with responses generated by a natural language processing model and an emotion engine. The program processing of the system is explained below step by step.

[1069] Overall system configuration

[1070] This system consists of a server, a terminal, and a user. The role of each component is explained in detail below.

[1071] 1. Data Collection

[1072] To start using the system, users enter and upload basic information about the deceased person and their LINE message history. For example, imagine a situation where a user enters the name and date of birth of the deceased person and uploads a text file of past LINE messages.

[1073] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server in a secure manner.

[1074] 2. Data storage

[1075] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[1076] 3. Data Learning

[1077] The server preprocesses the stored message history and trains a natural language processing model, which includes tokenizing the message text, removing unnecessary data, and morphological analysis. The preprocessed data is used to train the model on the deceased person's unique speaking and phrasing patterns.

[1078] 4. Generating dialogue with the user

[1079] A user uses a terminal to enter a message to initiate a conversation with the system. For example, a user might enter, "How was your day?"

[1080] The terminal sends the input message from the user to the server, with necessary metadata (such as a timestamp and user ID) added to the input message.

[1081] The server analyzes the message received from the user and generates an appropriate response using a trained natural language processing model. The emotion engine is also utilized here. Specifically, it recognizes the emotion in the user's input message and adjusts the tone and content of the response based on that emotion. For example, if the user says, "Today was a tough day," the emotion engine recognizes the "tough" emotion and adjusts the response to be more warmhearted.

[1082] The device displays the response received from the server to the user, allowing the user to experience the experience of interacting with the deceased.

[1083] 5. Ongoing dialogue and feedback

[1084] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[1085] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion engine. It analyzes the feedback data and retrains it to improve the quality of responses.

[1086] 6. Data Security

[1087] The server uses encryption methods for storing and transmitting collected data, which ensures the protection of user privacy.

[1088] Through these steps, users can ease their grief through an advanced dialogue experience. Furthermore, by utilizing an emotion engine, the system can generate responses that are in tune with the user's emotions, enabling more personalized care.

[1089] The processing flow will be explained below.

[1090] Step 1:

[1091] The user uses the device to enter basic information about the deceased person and their past message history and upload it. For example, the user enters the deceased person's name and date of birth, selects the LINE message history as a text file, and presses the send button.

[1092] Step 2:

[1093] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server using a secure communication protocol.

[1094] Step 3:

[1095] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[1096] Step 4:

[1097] The server preprocesses the stored message history, which includes tokenizing the message text and removing unnecessary data, and converts the formatted data into a format suitable for training natural language processing models.

[1098] Step 5:

[1099] The server uses the preprocessed data to train a natural language processing model. During the training process, the model learns the deceased person's unique speech and phrasing patterns. Through this process, the model learns the deceased person's linguistic style.

[1100] Step 6:

[1101] A user inputs a message through a terminal to start a conversation with the system, for example, "How was your day?", and presses the send button.

[1102] Step 7:

[1103] The device sends the input message to the server. The message is accompanied by necessary metadata (such as a timestamp and user ID) and is forwarded to the server while ensuring reliability.

[1104] Step 8:

[1105] The server analyzes messages received from users and uses an emotion engine to recognize the user's emotions. For example, if a user types "I had a hard time today," the emotion engine will recognize "sadness" or "stress" from the user's message.

[1106] Step 9:

[1107] The server uses a natural language processing model to generate an appropriate response based on the emotional data recognized by the emotion engine. The response is tailored to reflect the user's emotions. For example, in response to an input such as "Today was tough," a warm response such as "That must have been really tough. Thank you for talking to me" is generated.

[1108] Step 10:

[1109] The server then sends the generated response to the terminal, converting the response message into the required format so that it can be properly displayed to the user.

[1110] Step 11:

[1111] The device displays the responses received from the server to the user, allowing the user to experience the feeling of having a conversation with the deceased. Furthermore, the responses are emotionally sensitive, providing a deeper interaction experience.

[1112] Step 12:

[1113] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[1114] Step 13:

[1115] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion engine. The feedback data is analyzed and retrained to improve the quality of responses. This effort allows for continuous improvement of the overall system performance and user satisfaction.

[1116] This allows users to ease their grief through a sophisticated dialogue experience, allowing the system to provide responses that are sensitive to the user's emotions, while also ensuring data security and privacy.

[1117] Example 2

[1118] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1119] In modern society, people miss their deceased loved ones, but there are limited means to recreate their memories and messages. However, existing technologies have not yet fully developed systems that provide a dialogue experience with the deceased, making it difficult for users to experience the sensation of "talking" with the deceased. Therefore, new methods are needed to provide users with psychological comfort and healing by maintaining contact with the deceased.

[1120] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1121] In this invention, the server includes means for a user to input and upload basic information about the deceased and their communication history, means for securely processing the input and uploaded data and transferring it to the server, means for decompressing and decrypting the received data and storing it in a database, means for preprocessing the stored communication history, tokenizing and analyzing the data, and training a natural language processing model, means for receiving an input message from the user and generating a response using the trained natural language processing model and an emotion estimation engine, and means for displaying the generated response to the user, thereby enabling the user to have a conversation experience with the deceased.

[1122] "Deceased" refers to someone who has already died.

[1123] "Basic information" refers to the main data that can be used to identify an individual, such as name and date of birth.

[1124] "Communication history" refers to a record of past messages and conversations.

[1125] "User" refers to an individual who uses this system.

[1126] "Input and upload" refers to the act of a user providing data to the system.

[1127] "Secure processing" refers to taking measures such as compression and encryption to handle data safely.

[1128] "Server" refers to a computer system that stores and processes data received from users.

[1129] "Decompressing and decrypting" refers to restoring compressed data to its original form and decrypting encrypted data.

[1130] A "database" refers to a system for organizing and storing data.

[1131] "Preprocessing" refers to processes such as data cleaning and tokenization to make the data suitable for training a model.

[1132] "Tokenization" refers to the process of dividing text data into meaningful units.

[1133] A "natural language processing model" refers to a machine learning algorithm for understanding and generating human language.

[1134] An "emotion estimation engine" refers to an algorithm that analyzes and estimates emotions from input text data.

[1135] "Generating a reply" refers to creating an appropriate response to an input message.

[1136] "Display" refers to outputting the generated response to the user's terminal screen.

[1137] This invention is a system that provides a conversational experience with a deceased person. It collects basic information and communication history of the deceased person and provides the user with responses generated by a natural language processing model and an emotion estimation engine. The system is composed of a server, a terminal, and a user.

[1138] Data Collection and Transmission

[1139] To use the system, users must first enter and upload basic information and communication history of the deceased. Specifically, users enter the name and date of birth of the deceased and upload past communication history (e.g., text file format of messages).

[1140] The terminal receives this data, compresses and encrypts it, and transfers it to the server. For data compression, it uses the zip library, and for encryption, it uses the AES encryption library. The data is sent to the server using a secure communication protocol (for example, HTTPS).

[1141] Receiving and storing data

[1142] The server receives the encrypted data, unpacks and decrypts it using a zip library for unpacking and an AES decryption library for decryption. The server then stores the unpacked and decrypted data in a database (e.g., MySQL).

[1143] Data Learning

[1144] The server preprocesses the stored communication history and trains a natural language processing model, which includes tokenization and removal of unnecessary data using morphological analysis tools (e.g., MeCab), to learn the deceased person's unique speech patterns and phrasing.

[1145] User interaction

[1146] A user uses a terminal to enter a message to initiate a conversation with the system. For example, a user might enter, "How was your day?"

[1147] The terminal sends the input message from the user to the server, with metadata such as a timestamp and user ID attached to the message.

[1148] The server analyzes messages received from users and generates appropriate responses using a trained natural language processing model and an emotion estimation engine. For example, if a user says, "Today was a tough day," the emotion engine recognizes the "toughness" and adjusts the response to be more warm-hearted.

[1149] The device displays the response received from the server to the user, allowing the user to experience the experience of interacting with the deceased.

[1150] Continuous feedback and improvement

[1151] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[1152] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion estimation engine. It analyzes the feedback data and retrains it to improve the quality of responses.

[1153] Data Security

[1154] The server uses encryption methods when storing and transmitting collected data, thereby ensuring user privacy.

[1155] Examples of concrete examples and prompts

[1156] For example, if a user sends a message saying "How was your day?", the server will do the following:

[1157] 1. The user uses a terminal to type, "How was your day?"

[1158] 2. The device sends this message to the server.

[1159] 3. The server analyzes the message and uses an emotion engine to recognize the user's emotion.

[1160] 4. The trained natural language processing model generates a response, for example, "I was thinking of you today, how are you?"

[1161] 5. The server generates a response and sends it to the terminal.

[1162] 6. The terminal displays the reply to the user.

[1163] In this way, users can enjoy the interactive experience of interacting with the deceased, and continuous feedback improves the accuracy of the system.

[1164] Examples of prompts include the following:

[1165] User: How was your day?

[1166] Model: I was thinking of you today, how are you?

[1167] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1168] Step 1:

[1169] The user enters and uploads basic information and communication history of the deceased.

[1170] Input: Basic information of the deceased (e.g. name, date of birth), text file of communication history.

[1171] Specific operation: The user enters the name of the deceased in the name input field, the date of birth of the deceased in the date of birth field, selects the text file of the communication history using the file selection function, and presses the upload button.

[1172] Output: User-entered data sent to the terminal.

[1173] Step 2:

[1174] The terminal receives data entered and uploaded by the user, compresses and encrypts it, and transfers it to the server.

[1175] Input: User-entered data (basic information of the deceased, text file of communication history)

[1176] Specific operation: The terminal compresses the data using a data compression library (e.g., zip library), encrypts the compressed data using an encryption library (e.g., AES encryption), and transfers it to the server using the HTTPS protocol.

[1177] Output: The encrypted and compressed data sent to the server.

[1178] Step 3:

[1179] The server receives the encrypted data, decompresses and decrypts it, and stores it in a database.

[1180] Input: Encrypted and compressed data.

[1181] What happens: The server decrypts the data using an AES decryption library, unpacks it using a zip library, and then stores the unpacked data appropriately in the database system (e.g., MySQL).

[1182] Output: The decompressed and decrypted data is stored in the database.

[1183] Step 4:

[1184] The server preprocesses the stored communication history and trains a natural language processing model.

[1185] Input: The deceased person's communication history stored in a database.

[1186] What it does: The server uses a natural language processing library (e.g., NLTK, spaCy) to tokenize the message text and remove unnecessary data. The preprocessed data is used to train a model to learn the unique speaking style and phrasing of the deceased.

[1187] Output: A trained natural language processing model.

[1188] Step 5:

[1189] The user uses the terminal to enter messages to interact with the system.

[1190] Input: A message typed by the user (e.g., "How was your day?")

[1191] Specific operation: The user enters a message in the input form and presses the send button.

[1192] Output: User input messages sent to the terminal.

[1193] Step 6:

[1194] The terminal sends an input message from the user to the server.

[1195] Input: User input message.

[1196] Specific operation: The terminal adds metadata such as a timestamp and user ID to the input message and sends it to the server using the HTTPS protocol.

[1197] Output: The user's input message and metadata sent to the server.

[1198] Step 7:

[1199] The server analyzes messages received from users and generates appropriate responses using a trained natural language processing model and emotion estimation engine.

[1200] Input: The message and metadata sent by the user.

[1201] How it works: The server analyzes the message content using a natural language processing library and identifies the user's sentiment using a sentiment analysis library (e.g., TextBlob, VADER). A trained generative AI model generates a response, and the sentiment estimation engine adjusts the response.

[1202] Output: The generated reply message.

[1203] Step 8:

[1204] The terminal displays the response received from the server to the user.

[1205] Input: The reply message sent by the server.

[1206] Specific operation: The terminal displays the received reply message on the screen, visually presenting it to the user.

[1207] Output: The reply message that is displayed to the user.

[1208] Step 9:

[1209] The user provides feedback on the system's response.

[1210] Input: User ratings and comments.

[1211] Specific operation: The user enters the response rating and comments in the feedback form and presses the submit button.

[1212] Output: Feedback data sent to the device.

[1213] Step 10:

[1214] The server collects feedback from users and reflects it in improving the natural language processing model and emotion estimation engine.

[1215] Input: Feedback data from users.

[1216] What it does: The server stores the feedback data in a database, analyzes it to identify areas for improvement, and retrains the model.

[1217] Output: Improved natural language processing models and sentiment estimation engines.

[1218] (Application example 2)

[1219] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1220] Conventional dialogue systems have difficulty providing a dialogue experience with the deceased, and lack emotional care when users interact with the deceased through past message history. Furthermore, there is no function to provide support for products related to the deceased within the virtual store, making it difficult for users to select products that are considerate to the deceased. Our goal is to solve these issues and provide a more personal and emotionally sensitive dialogue experience.

[1221] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1222] In this invention, the server includes means for collecting and storing basic information and communication history of the deceased, means for preprocessing the collected communication history to train a natural language processing model, means for receiving an input message from a user and generating a response using the trained natural language processing model, means for displaying the generated response to the user, means for providing support for inquiries about products related to the deceased in a virtual store, and means for adjusting the tone and content of the response using an emotion engine. This enables the user to purchase products related to the deceased in the virtual store while receiving personal support through a dialogue with the deceased.

[1223] "Deceased" refers to a person who has passed away.

[1224] "Basic information" refers to personal data such as the deceased's name and date of birth.

[1225] "Communication history" refers to past messages and conversation records between the deceased and the user.

[1226] A "natural language processing model" is an algorithm that understands and processes human language, generating responses based on input text.

[1227] "Preprocessing" refers to the process of preparing collected data in a form that is easy to analyze by tokenizing it, removing unnecessary data, performing morphological analysis, etc.

[1228] "User" refers to a person who uses this system to have an interactive experience with a deceased person.

[1229] "Input message" refers to a text message that a user sends to the system.

[1230] "Reply" refers to a response message generated by the system that mimics the speech style and emotions of the deceased.

[1231] "Display means" refers to a method or device for displaying the generated response on the user's terminal.

[1232] A "virtual store" refers to a virtual store space where products and services can be purchased online.

[1233] An "emotion engine" is an algorithm that analyzes emotions from a user's input message and adjusts the tone and content of the response based on those emotions.

[1234] The present invention is a system that allows users to have a conversational experience with a deceased person. It collects and stores basic information and communication history of the deceased, and can train a natural language processing model based on that data. Furthermore, it has the function of providing support for inquiries about products related to the deceased in a virtual store.

[1235] Overall system configuration

[1236] This system is composed of a server, a terminal, and a user, and operates as follows.

[1237] Data collection and storage

[1238] To start using the system, users input and upload basic information about the deceased person and their communication history, such as their name, date of birth, and past message history in text format. The device receives this information, compresses and encrypts the data, and sends it to the server.

[1239] The server decompresses and decrypts the received data and stores it in a database. When storing it, it properly associates the deceased person's basic information and message history with the data and stores it in secure storage, thereby protecting the user's privacy.

[1240] Data preprocessing and model training

[1241] The server preprocesses the stored message history, which includes tokenizing the messages, removing unnecessary data, and morphological analysis. The preprocessed data is then used to train a natural language processing model to learn the deceased person's unique phrasing and speaking patterns.

[1242] User interaction generation and display

[1243] The user uses the device to input a message to initiate a conversation with the system. For example, they could ask, "Did the deceased have any special memories of this accessory?" The device then sends this input message to the server, which then analyzes the message and generates an appropriate response using a trained natural language processing model. The server also uses an emotion engine to recognize emotions in the user's input message and adjust the tone and content of the response based on those emotions.

[1244] The device displays the generated response to the user, allowing the user to experience as if they were having a conversation with the deceased person.

[1245] Support functions within the virtual store

[1246] The system includes a function that provides support for inquiries about products related to the deceased in the virtual store. For example, when a user finds an accessory related to the deceased in the virtual store, they can ask about the deceased's memories and stories associated with the accessory and receive appropriate answers. This allows users to select products through a more personalized experience.

[1247] Software and hardware used

[1248] Server: Used for storing data, preprocessing, training natural language processing models, and generating responses.

[1249] Device: The device (smartphone, tablet, etc.) that a user uses to type and receive messages.

[1250] Natural language processing model: Using OpenAI GPT-3 and other models, responses are generated based on the user's input message.

[1251] Emotion engine: Analyzes the sentiment of a user's message and adjusts the tone and content of responses accordingly.

[1252] Examples of concrete examples and prompts

[1253] For example, if a user is searching for an accessory related to a deceased person in a virtual store, a possible question might be, "Did the deceased have any special memories associated with this accessory?"

[1254] Example prompt sentence:

[1255] How was your day?

[1256] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1257] Step 1:

[1258] Data collection

[1259] Users input and upload basic information and communication history of the deceased person into the device. Specifically, they provide the deceased person's name, date of birth, and past message history as a text file. The device receives this data, compresses and encrypts it, and sends it to the server.

[1260] Input: Deceased person's name, date of birth, message history.

[1261] Output: Compressed and encrypted data.

[1262] Step 2:

[1263] Data storage

[1264] The server receives the data sent from the device, decompresses and decrypts it, and then associates the deceased person's basic information and message history with a database and stores them in a secure storage environment to ensure the data remains private.

[1265] Input: Compressed and encrypted data.

[1266] Output: Basic information and message history of the deceased person stored in the database.

[1267] Step 3:

[1268] Data Preprocessing

[1269] The server preprocesses the stored message history by tokenizing the messages, removing unnecessary data, and performing morphological analysis. This process prepares the data in a format that is easy to analyze.

[1270] Input: The saved message history.

[1271] Output: Preprocessed data.

[1272] Step 4:

[1273] Training the model

[1274] The server uses the preprocessed data to train a natural language processing model, which learns the specific phrasing and speech patterns of the deceased.

[1275] Input: Preprocessed data.

[1276] Output: A trained natural language processing model.

[1277] Step 5:

[1278] Dialogue Generation

[1279] The user inputs a message to the system using the terminal, for example, a question such as "Did the deceased have any special memories about this accessory?" The terminal then sends this input message to the server.

[1280] Input: The user's input message.

[1281] Output: The input message sent to the server.

[1282] Step 6:

[1283] Response Generation

[1284] The server analyzes messages sent by users and generates appropriate responses using a trained natural language processing model, including an emotion engine to analyze the user's emotions and adjust the tone and content of the response.

[1285] Input: The input message sent to the server.

[1286] Output: The generated response.

[1287] Step 7:

[1288] Viewing the response

[1289] The terminal displays the generated response received from the server to the user, thereby allowing the user to have a conversational experience with the deceased person.

[1290] Input: The generated response.

[1291] Output: The response displayed to the user.

[1292] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1293] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1294] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1295] [Fourth embodiment]

[1296] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1297] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1298] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1299] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1300] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1301] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1302] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1303] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1304] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1305] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1306] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1307] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1308] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1309] The present invention is a system that provides a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing the user with responses generated by a natural language processing model. The program processing of the system is explained below in order.

[1310] Overall system configuration

[1311] This system mainly consists of a server, a terminal, and a user. The roles of each component are as follows:

[1312] 1. Data Collection

[1313] To start using the system, the user provides basic information about the deceased person (such as name and date of birth) and their LINE message history. As a concrete example, we will assume a situation where the user uploads the deceased person's past LINE messages to their device as a text file.

[1314] The server receives the uploaded data and stores it in a secure database, converts it into a suitable format, and stores it encrypted to protect privacy.

[1315] 2. Data Learning

[1316] The server preprocesses the stored message history to train a natural language processing model. Preprocessing includes, for example, tokenizing the message text and removing unnecessary noise. For example, the server analyzes the grammar and syntax of LINE messages to create a dataset for machine learning.

[1317] The server uses natural language processing techniques to train the model to recognize the deceased person's unique speaking style and expressions. During the training process, patterns are extracted from message history and reflected in the model. For example, the model learns the deceased person's frequently used phrases and unique expressions.

[1318] 3. Generating dialogue with the user

[1319] The user enters a message at the terminal to initiate a dialogue with the system. The user can enter, for example, "How was your day?"

[1320] The terminal sends input from the user to the server. At this time, the terminal adds necessary metadata (e.g., a timestamp) to the input message and sends it to the server. As a concrete example, the terminal sends the user's message "How was your day?" to the server.

[1321] The server generates a response based on the message received from the user, using a trained natural language processing model to generate a response that the deceased person would likely respond with, for example, "I may not have done much today, but I'm always thinking of you."

[1322] The terminal displays the response received from the server to the user, allowing the user to have an experience that feels as if they are interacting with the deceased.

[1323] 4. Ongoing dialogue and feedback

[1324] The user can provide feedback on the system's response, for example by being presented with a button to rate whether the response was appropriate.

[1325] The server improves the natural language processing model based on user feedback. The collected feedback information is used to further train the model. For example, the accuracy of the model can be improved by relearning response patterns where users have given positive feedback.

[1326] This allows users to ease their grief through conversations with the deceased. Data privacy is also ensured, making this a system that users can use with peace of mind.

[1327] The processing flow will be explained below.

[1328] Step 1:

[1329] The user uses the device to enter and upload the deceased person's basic information and LINE message history. Specifically, the user enters the deceased person's name and date of birth into the form, selects a text file of past LINE messages, and presses the send button.

[1330] Step 2:

[1331] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server in a secure manner.

[1332] Step 3:

[1333] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[1334] Step 4:

[1335] The server preprocesses the stored message history, including tokenizing the message text, removing unnecessary data, and performing morphological analysis, so that the formatted data is suitable for training natural language processing models.

[1336] Step 5:

[1337] The server uses the preprocessed data to train a natural language processing model. During the training process, the model learns the deceased person's unique speech patterns and phrasing. Specifically, a recurrent neural network and a transformer model are used to build a language model of the deceased person.

[1338] Step 6:

[1339] A user uses a terminal to enter a message to initiate a conversation with the system, for example, "How was your day?", into an input field on the terminal and presses the send button.

[1340] Step 7:

[1341] The terminal receives an input message from the user and sends it to the server, adding necessary metadata (such as a timestamp or user ID) to the input message and ensuring its reliability.

[1342] Step 8:

[1343] The server analyzes messages received from users and uses a trained natural language processing model to generate appropriate responses, which are naturally crafted based on the linguistic style of the deceased.

[1344] Step 9:

[1345] The server then sends the generated response to the terminal, converting the response message into the required format so that it can be properly displayed to the user.

[1346] Step 10:

[1347] The terminal displays the response received from the server to the user, who can then experience the experience of interacting with the deceased person.

[1348] Step 11:

[1349] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[1350] Step 12:

[1351] The server stores the feedback collected from users and reflects it in improving the natural language processing model, analyzing the feedback data and retraining it to improve the quality of responses.

[1352] These steps allow users to ease their grief through an enhanced interactive experience, while ensuring data security and privacy.

[1353] Example 1

[1354] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1355] Currently, there are systems that provide users who want to reminisce about their deceased loved ones with an experience that makes it seem as if they are having a conversation with the deceased. However, these systems do not adequately address issues such as the security of the information provided by the user, continuous improvement of the model based on user feedback, and learning and reproducing the unique speaking style and expressions of the deceased. This makes it difficult for users to obtain a natural and realistic conversational experience.

[1356] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1357] In this invention, the server includes means for collecting and storing basic information and message history of the deceased, means for preprocessing the collected message history to train a natural language processing model, means for receiving an input message from a user and generating a reply using the trained natural language processing model, means for displaying the generated reply to the user, and means for adding metadata such as a timestamp to the input message from the user and transmitting it to the server. This makes it possible to securely store data, improve the model using feedback, and reproduce the deceased's unique speaking style and expressions, thereby providing the user with a more natural and realistic conversation experience.

[1358] "Basic information about the deceased" refers to the name, date of birth, and other basic information needed to identify the deceased.

[1359] "Message history" refers to records of text messages and chats exchanged by a deceased person during their lifetime, and is primarily used as training data for natural language processing models.

[1360] A "natural language processing model" is a machine learning algorithm designed to understand and generate human language, generating appropriate responses to messages input by users.

[1361] "Tokenization" is a preprocessing technique that divides text data into smaller units such as words or phrases.

[1362] "Preprocessing" refers to a series of processes that remove unnecessary noise from collected data and prepare it in a format suitable for training a natural language processing model.

[1363] A "timestamp" is information indicating the date and time when a specific event occurred, and is added to a message input by a user.

[1364] "Feedback" refers to the user's evaluation or opinion on the system's response, and is data used to improve the model.

[1365] A "trained natural language processing model" is a model that has learned based on the deceased person's message history and is capable of reproducing the deceased person's unique speaking style and phrasing.

[1366] "Encryption" is the process of converting collected data using a specific algorithm so that it cannot be easily analyzed or viewed by third parties.

[1367] A "database" is a system for systematically storing and managing collected data.

[1368] The present invention provides a system for providing a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing a user with responses generated by a natural language processing model. Specific embodiments of the system are described below.

[1369] The hardware required to implement this system is a server, user terminals, and a network connection. The software includes a database management system for data collection and storage, a natural language processing model, and encryption algorithms.

[1370] Data collection and storage

[1371] To begin using the system, users first provide basic information about the deceased (such as name and date of birth) and their message history. This information is primarily in the form of a text file, and users upload it to the system using their device. Specifically, users prepare LINE messages with the deceased as a text file and upload it to their device.

[1372] The device collects the information uploaded by the user and sends it to the server, which stores the received information in a database. At this time, the data is converted into an appropriate format, encrypted, and stored securely.

[1373] Data Preprocessing and Training

[1374] The server pre-processes the stored message history, which includes tokenizing the messages and removing unnecessary noise. For example, the server removes special characters and unnecessary whitespace from the message history and tokenizes it into words.

[1375] The pre-processed data is used to train a natural language processing model to learn the distinctive speech patterns and phrasing of the deceased, which the server uses to build the model and extract patterns and features from the messages.

[1376] User interaction generation

[1377] A user types a message at a terminal to initiate a dialogue with the system. For example, the user types "How was your day?"

[1378] The device sends the message entered by the user, along with metadata such as a timestamp, to the server. The server then analyzes the message and uses a trained natural language processing model to generate a response, such as, "I may not have done much today, but I'm always thinking of you."

[1379] The device displays the generated response to the user, allowing the user to experience as if they were interacting with the deceased.

[1380] Feedback and Improvements

[1381] The user provides feedback on the system's response by providing a button to rate whether the response is appropriate or not.

[1382] The server improves the natural language processing model based on user feedback. By collecting feedback data and retraining the model based on it, it becomes possible to generate more accurate responses.

[1383] Specific examples

[1384] For example, if a user wishes to reminisce about a deceased loved one, they could write a prompt like this:

[1385] "What was your favorite movie?"

[1386] "Tell me about the last trip you took together."

[1387] As the user enters these prompts, the system infers how the deceased would have answered these questions and generates an appropriate response, such as, "My favorite movie was 'Favorite Movie Title.' I always enjoyed watching it with you."

[1388] Through this system, users can enjoy the experience of interacting with the deceased and ease the grief of bereavement.In addition, data privacy is ensured, so users can use the system with peace of mind.

[1389] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1390] Step 1:

[1391] Data collection

[1392] To start using the system, the user uploads the deceased's basic information (such as name and date of birth) and the message history with the deceased to the device. Specifically, the user provides the LINE messages with the deceased in the form of a text file to the device. The input is the deceased's basic information and message history text file, and the output is the input data saved on the device.

[1393] The terminal reads the uploaded text file. Specifically, the terminal loads the file contents into memory, checks the data format, and then sends the data to the server.

[1394] Step 2:

[1395] Data storage

[1396] The server securely receives the data sent from the device and stores it in a database. At this time, the data is converted into an easy-to-analyze format and encrypted. The input is the basic information and message history of the deceased person sent from the device, and the output is securely stored in the database.

[1397] Specifically, the server stores basic information about the deceased in specific fields, converts message history from plain text to JSON format, and encrypts the data using an encryption algorithm to protect privacy.

[1398] Step 3:

[1399] Data Preprocessing

[1400] The server preprocesses the stored message history, which includes tokenization and removal of unnecessary noise. The input is the message history data retrieved from the database, and the output is the preprocessed dataset.

[1401] Specifically, the server removes special characters and unnecessary spaces from the message history, processes it into a grammatically separable form, tokenizes the data, and splits it into words or phrases. Furthermore, during the noise removal process, it removes things like stop verbs, which are generally considered meaningless.

[1402] Step 4:

[1403] Training a natural language processing model

[1404] The server uses the preprocessed dataset to train a natural language processing model, where the input is the preprocessed dataset and the output is the trained natural language processing model.

[1405] Specifically, the server runs machine learning algorithms to extract patterns and features from the message history, training the model to recognize the unique speech patterns and phrasing of the deceased, fine-tuning specific phrases and unique expressions.

[1406] Step 5:

[1407] User interaction generation

[1408] A user enters a message at a terminal to initiate a conversation with the system, for example, entering a prompt such as "How was your day?" The input is the message from the user, and the output is the message sent to the server.

[1409] The terminal sends the message entered by the user to the server, with metadata such as a timestamp attached to the message. Specifically, the terminal formats the message, adds a timestamp, and sends it to the server over the network.

[1410] The server analyzes the received message and generates a response using a trained natural language processing model. The input is the user's message, and the output is the generated response. For example, the server might generate a response like, "I may not have done much today, but I'm always thinking of you."

[1411] The device receives the generated response and displays it to the user, allowing the user to experience the experience as if they were interacting with the deceased person.

[1412] Step 6:

[1413] Feedback and model improvement

[1414] The user provides feedback on the system's response, for example by clicking a button to rate whether the response was appropriate. The input is the feedback from the user, and the output is the feedback data to the server.

[1415] The terminal transmits the user's feedback to the server, specifically, the terminal collects the feedback information and transmits it together with the metadata to the server.

[1416] The server improves the natural language processing model based on user feedback. The input is the feedback data, and the output is an improved natural language processing model. Specifically, the server uses the feedback to retrain the model and improve the accuracy of responses.

[1417] (Application example 1)

[1418] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1419] Many people seek peace of mind through conversations with their deceased loved ones. However, current technology makes it difficult to provide such experiences in real time. Furthermore, systems that provide such conversational experiences are typically limited to home or personal devices, and are rarely available in commercial facilities or brick-and-mortar stores. This presents a challenge in providing valuable experiences for customers, contributing to attracting more customers and improving customer satisfaction.

[1420] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1421] In this invention, the server includes means for collecting and storing basic information and message history of the deceased, means for preprocessing the collected message history to train a natural language processing model, and means for receiving input messages from users and generating responses using the trained natural language processing model. This allows users to have the experience of interacting with the deceased through an application program installed on a terminal in a physical store. This allows users to enjoy a special conversation experience based on their memories of the deceased in real time at commercial facilities or physical stores, which is expected to increase customer attraction and customer satisfaction.

[1422] "Basic information about the deceased" refers to basic personal information about the deceased, such as name, date of birth, and gender.

[1423] A "message history" is a record of text messages exchanged between the user and the deceased person.

[1424] "Collection" is the process of obtaining the required data from the user and storing that data.

[1425] "Preprocessing" is the process of analyzing raw data and converting it into a format suitable for training a natural language processing model.

[1426] A "natural language processing model" is a machine learning model designed to understand and generate human language.

[1427] "Training" is the process of training a natural language processing model using collected data.

[1428] A "trained natural language processing model" is a natural language processing model that has completed learning through collected data.

[1429] A "user input message" is a text message that a user inputs to the system.

[1430] A "reply" is a response message generated by a natural language processing model in response to an input message.

[1431] "Display" is the act of visually presenting the generated response to the user.

[1432] "Brick-and-mortar terminals" are communication terminals such as displays and robots installed within commercial facilities.

[1433] An "application program" is software that runs on a terminal in a physical store and provides users with an interactive experience.

[1434] "Feedback" refers to the evaluation or opinion that a user gives about the system's response.

[1435] "Encryption" is the process of transforming data using a specific algorithm in order to store it securely.

[1436] This invention is a system that collects basic information and message history of the deceased, and provides an experience where users can interact with the deceased based on this information. The system is mainly composed of a server, a terminal, and a user.

[1437] Server Roles

[1438] The server performs the following main functions:

[1439] 1. Data collection and storage: The user uploads the deceased person's basic information (such as name and date of birth) and message history (LINE and text messages, etc.). The server receives this data, encrypts it, and stores it in a secure database (e.g., PostgreSQL).

[1440] 2. Model Preprocessing and Training: The collected message history is preprocessed for the natural language processing model (e.g., OpenAI's GPT-4) to be used. Preprocessing involves tokenizing the message text and removing noise. The server then uses the preprocessed data to train the natural language processing model to learn the deceased's unique speaking style and phrasing.

[1441] Device Role

[1442] The devices installed in physical stores (e.g. smart displays and robots) have the following functions:

[1443] 1. Receiving a message from the user: A message entered by the user through the terminal is sent to the server. For example, the user enters "Mom, how was your day?"

[1444] 2. Receiving and displaying a response: The generated response received from the server is displayed to the user. For example, the server generates a response such as "I may not have done much today, but I'm always thinking of you" and sends it to the terminal.

[1445] User Roles

[1446] A user uses the system as follows:

[1447] 1. Providing information about the deceased: Provide basic information and message history about the deceased when starting the system.

[1448] 2. Start of dialogue: Enter a message into a terminal in the physical store and receive a response from the server.

[1449] 3. Providing feedback: Providing feedback on the server-generated responses and contributing to the improvement of the system.

[1450] Sample prompt sentence

[1451] Based on the user's message entered on the terminal, the server generates a prompt and creates a response. For example, the following prompt is used:

[1452] User message:

[1453] "Mom, I want to cook your favorite dish. What was your favorite?"

[1454] Example prompt sentence:

[1455] Message History

[1456] User: Mom, I want to cook your favorite dish. What was your favorite?

[1457] Deceased:

[1458] Hardware and software used

[1459] Server: PostgreSQL is used as the database management system, and OpenAI's GPT-4 is used as the natural language processing model.

[1460] Terminals: Smart displays and robots are used to enable real-time interaction with users.

[1461] Secure storage: AES (Advanced Encryption Standard) encryption technology is used as the data encryption method.

[1462] This allows users to enjoy interacting with deceased loved ones even in physical stores, and the quality of the interaction experience can be improved by improving the model based on the feedback provided.

[1463] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1464] Step 1:

[1465] The user collects basic information and message history of the deceased person and uploads it to the system. Specifically, the user sends text files and LINE message history to the server via their device. The input is the deceased person's information and message history, and the output is the transmission of this data to the server.

[1466] Step 2:

[1467] The server receives the collected basic information and message history of the deceased and stores it in a secure database. Specifically, after receiving the data, it encrypts it using AES encryption technology and stores it in a PostgreSQL database. The input is the data sent by the user, and the output is stored in the database in encrypted form.

[1468] Step 3:

[1469] The server preprocesses the stored message history by tokenizing the message text and removing unnecessary noise (e.g., emojis and special symbols). The input is the message history retrieved from the database, and the output is the tokenized and preprocessed text data.

[1470] Step 4:

[1471] The server uses the preprocessed data to train a natural language processing model. Specifically, it uses the preprocessed dataset to train OpenAI's GPT-4 model to learn the deceased's unique speaking style and phrasing. The input is the preprocessed text data, and the output is the trained natural language processing model.

[1472] Step 5:

[1473] The user initiates a conversation with the deceased through a terminal in a physical store. Specifically, the user enters a message into the terminal and presses the "send" button. The input is the message the user enters into the terminal, and the output is the message being sent to the server.

[1474] Step 6:

[1475] The terminal sends the input message from the user to the server. Specifically, it adds metadata such as a timestamp and user ID to the input message and sends it to the server via the API. The input is the user's input message and metadata, and the output is that the message is sent to the server.

[1476] Step 7:

[1477] The server generates a response using the input message received from the user, specifically by generating a prompt sentence like the following and feeding it to the GPT-4 model:

[1478] Message History

[1479] User: Mom, I want to cook your favorite dish. What was your favorite?

[1480] Deceased:

[1481] The input is the user's input message and prompt, and the output is the generated response.

[1482] Step 8:

[1483] The server returns the generated response to the terminal. Specifically, it sends a response message to the terminal through the API. The input is the generated response, and the output is the response message returned to the terminal.

[1484] Step 9:

[1485] The terminal displays the response received from the server to the user. Specifically, a smart display or robot conveys the response message to the user visually or audibly. The input is the response message sent from the server, and the output is the response displayed to the user.

[1486] Step 10:

[1487] The user provides feedback on the system-generated response by pressing a button to rate whether the response was appropriate. The input is the user's rating, and the output is that rating is sent to the server.

[1488] Step 11:

[1489] The server collects user feedback and uses it to improve the natural language processing model. Specifically, it analyzes the feedback information and uses it to retrain the model. The input is the user feedback, and the output is an improved natural language processing model.

[1490] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1491] The present invention is a system that provides a conversational experience with a deceased person by collecting basic information and message history of the deceased person and providing the user with responses generated by a natural language processing model and an emotion engine. The program processing of the system is explained below step by step.

[1492] Overall system configuration

[1493] This system consists of a server, a terminal, and a user. The role of each component is explained in detail below.

[1494] 1. Data Collection

[1495] To start using the system, users enter and upload basic information about the deceased person and their LINE message history. For example, imagine a situation where a user enters the name and date of birth of the deceased person and uploads a text file of past LINE messages.

[1496] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server in a secure manner.

[1497] 2. Data storage

[1498] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[1499] 3. Data Learning

[1500] The server preprocesses the stored message history and trains a natural language processing model, which includes tokenizing the message text, removing unnecessary data, and morphological analysis. The preprocessed data is used to train the model on the deceased person's unique speaking and phrasing patterns.

[1501] 4. Generating dialogue with the user

[1502] A user uses a terminal to enter a message to initiate a conversation with the system. For example, a user might enter, "How was your day?"

[1503] The terminal sends the input message from the user to the server, with necessary metadata (such as a timestamp and user ID) added to the input message.

[1504] The server analyzes the message received from the user and generates an appropriate response using a trained natural language processing model. The emotion engine is also utilized here. Specifically, it recognizes the emotion in the user's input message and adjusts the tone and content of the response based on that emotion. For example, if the user says, "Today was a tough day," the emotion engine recognizes the "tough" emotion and adjusts the response to be more warmhearted.

[1505] The device displays the response received from the server to the user, allowing the user to experience the experience of interacting with the deceased.

[1506] 5. Ongoing dialogue and feedback

[1507] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[1508] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion engine. It analyzes the feedback data and retrains it to improve the quality of responses.

[1509] 6. Data Security

[1510] The server uses encryption methods for storing and transmitting collected data, which ensures the protection of user privacy.

[1511] Through these steps, users can ease their grief through an advanced dialogue experience. Furthermore, by utilizing an emotion engine, the system can generate responses that are in tune with the user's emotions, enabling more personalized care.

[1512] The processing flow will be explained below.

[1513] Step 1:

[1514] The user uses the device to enter basic information about the deceased person and their past message history and upload it. For example, the user enters the deceased person's name and date of birth, selects the LINE message history as a text file, and presses the send button.

[1515] Step 2:

[1516] The device receives the entered and uploaded data and sends it to the server, ensuring that the data is compressed and encrypted before being transferred to the server using a secure communication protocol.

[1517] Step 3:

[1518] The server unpacks and decrypts the received data and stores it in a database, properly associating it with the deceased person's basic information and message history, and storing it in secure storage.

[1519] Step 4:

[1520] The server preprocesses the stored message history, which includes tokenizing the message text and removing unnecessary data, and converts the formatted data into a format suitable for training natural language processing models.

[1521] Step 5:

[1522] The server uses the preprocessed data to train a natural language processing model. During the training process, the model learns the deceased person's unique speech and phrasing patterns. Through this process, the model learns the deceased person's linguistic style.

[1523] Step 6:

[1524] A user inputs a message through a terminal to start a conversation with the system, for example, "How was your day?", and presses the send button.

[1525] Step 7:

[1526] The device sends the input message to the server. The message is accompanied by necessary metadata (such as a timestamp and user ID) and is forwarded to the server while ensuring reliability.

[1527] Step 8:

[1528] The server analyzes messages received from users and uses an emotion engine to recognize the user's emotions. For example, if a user types "I had a hard time today," the emotion engine will recognize "sadness" or "stress" from the user's message.

[1529] Step 9:

[1530] The server uses a natural language processing model to generate an appropriate response based on the emotional data recognized by the emotion engine. The response is tailored to reflect the user's emotions. For example, in response to an input such as "Today was tough," a warm response such as "That must have been really tough. Thank you for talking to me" is generated.

[1531] Step 10:

[1532] The server then sends the generated response to the terminal, converting the response message into the required format so that it can be properly displayed to the user.

[1533] Step 11:

[1534] The device displays the responses received from the server to the user, allowing the user to experience the feeling of having a conversation with the deceased. Furthermore, the responses are emotionally sensitive, providing a deeper interaction experience.

[1535] Step 12:

[1536] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[1537] Step 13:

[1538] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion engine. The feedback data is analyzed and retrained to improve the quality of responses. This effort allows for continuous improvement of the overall system performance and user satisfaction.

[1539] This allows users to ease their grief through a sophisticated dialogue experience, allowing the system to provide responses that are sensitive to the user's emotions, while also ensuring data security and privacy.

[1540] Example 2

[1541] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1542] In modern society, people miss their deceased loved ones, but there are limited means to recreate their memories and messages. However, existing technologies have not yet fully developed systems that provide a dialogue experience with the deceased, making it difficult for users to experience the sensation of "talking" with the deceased. Therefore, new methods are needed to provide users with psychological comfort and healing by maintaining contact with the deceased.

[1543] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1544] In this invention, the server includes means for a user to input and upload basic information about the deceased and their communication history, means for securely processing the input and uploaded data and transferring it to the server, means for decompressing and decrypting the received data and storing it in a database, means for preprocessing the stored communication history, tokenizing and analyzing the data, and training a natural language processing model, means for receiving an input message from the user and generating a response using the trained natural language processing model and an emotion estimation engine, and means for displaying the generated response to the user, thereby enabling the user to have a conversation experience with the deceased.

[1545] "Deceased" refers to someone who has already died.

[1546] "Basic information" refers to the main data that can be used to identify an individual, such as name and date of birth.

[1547] "Communication history" refers to a record of past messages and conversations.

[1548] "User" refers to an individual who uses this system.

[1549] "Input and upload" refers to the act of a user providing data to the system.

[1550] "Secure processing" refers to taking measures such as compression and encryption to handle data safely.

[1551] "Server" refers to a computer system that stores and processes data received from users.

[1552] "Decompressing and decrypting" refers to restoring compressed data to its original form and decrypting encrypted data.

[1553] A "database" refers to a system for organizing and storing data.

[1554] "Preprocessing" refers to processes such as data cleaning and tokenization to make the data suitable for training a model.

[1555] "Tokenization" refers to the process of dividing text data into meaningful units.

[1556] A "natural language processing model" refers to a machine learning algorithm for understanding and generating human language.

[1557] An "emotion estimation engine" refers to an algorithm that analyzes and estimates emotions from input text data.

[1558] "Generating a reply" refers to creating an appropriate response to an input message.

[1559] "Display" refers to outputting the generated response to the user's terminal screen.

[1560] This invention is a system that provides a conversational experience with a deceased person. It collects basic information and communication history of the deceased person and provides the user with responses generated by a natural language processing model and an emotion estimation engine. The system is composed of a server, a terminal, and a user.

[1561] Data Collection and Transmission

[1562] To use the system, users must first enter and upload basic information and communication history of the deceased. Specifically, users enter the name and date of birth of the deceased and upload past communication history (e.g., text file format of messages).

[1563] The terminal receives this data, compresses and encrypts it, and transfers it to the server. For data compression, it uses the zip library, and for encryption, it uses the AES encryption library. The data is sent to the server using a secure communication protocol (for example, HTTPS).

[1564] Receiving and storing data

[1565] The server receives the encrypted data, unpacks and decrypts it using a zip library for unpacking and an AES decryption library for decryption. The server then stores the unpacked and decrypted data in a database (e.g., MySQL).

[1566] Data Learning

[1567] The server preprocesses the stored communication history and trains a natural language processing model, which includes tokenization and removal of unnecessary data using morphological analysis tools (e.g., MeCab), to learn the deceased person's unique speech patterns and phrasing.

[1568] User interaction

[1569] A user uses a terminal to enter a message to initiate a conversation with the system. For example, a user might enter, "How was your day?"

[1570] The terminal sends the input message from the user to the server, with metadata such as a timestamp and user ID attached to the message.

[1571] The server analyzes messages received from users and generates appropriate responses using a trained natural language processing model and an emotion estimation engine. For example, if a user says, "Today was a tough day," the emotion engine recognizes the "toughness" and adjusts the response to be more warm-hearted.

[1572] The device displays the response received from the server to the user, allowing the user to experience the experience of interacting with the deceased.

[1573] Continuous feedback and improvement

[1574] The user can provide feedback on the system's response, for example by using a button to rate whether the response was appropriate.

[1575] The server stores the feedback collected from users and reflects it in improving the natural language processing model and emotion estimation engine. It analyzes the feedback data and retrains it to improve the quality of responses.

[1576] Data Security

[1577] The server uses encryption methods when storing and transmitting collected data, thereby ensuring user privacy.

[1578] Examples of concrete examples and prompts

[1579] For example, if a user sends a message saying "How was your day?", the server will do the following:

[1580] 1. The user uses a terminal to type, "How was your day?"

[1581] 2. The device sends this message to the server.

[1582] 3. The server analyzes the message and uses an emotion engine to recognize the user's emotion.

[1583] 4. The trained natural language processing model generates a response, for example, "I was thinking of you today, how are you?"

[1584] 5. The server generates a response and sends it to the terminal.

[1585] 6. The terminal displays the reply to the user.

[1586] In this way, users can enjoy the interactive experience of interacting with the deceased, and continuous feedback improves the accuracy of the system.

[1587] Examples of prompts include the following:

[1588] User: How was your day?

[1589] Model: I was thinking of you today, how are you?

[1590] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1591] Step 1:

[1592] The user enters and uploads basic information and communication history of the deceased.

[1593] Input: Basic information of the deceased (e.g. name, date of birth), text file of communication history.

[1594] Specific operation: The user enters the name of the deceased in the name input field, the date of birth of the deceased in the date of birth field, selects the text file of the communication history using the file selection function, and presses the upload button.

[1595] Output: User-entered data sent to the terminal.

[1596] Step 2:

[1597] The terminal receives data entered and uploaded by the user, compresses and encrypts it, and transfers it to the server.

[1598] Input: User-entered data (basic information of the deceased, text file of communication history)

[1599] Specific operation: The terminal compresses the data using a data compression library (e.g., zip library), encrypts the compressed data using an encryption library (e.g., AES encryption), and transfers it to the server using the HTTPS protocol.

[1600] Output: The encrypted and compressed data sent to the server.

[1601] Step 3:

[1602] The server receives the encrypted data, decompresses and decrypts it, and stores it in a database.

[1603] Input: Encrypted and compressed data.

[1604] What happens: The server decrypts the data using an AES decryption library, unpacks it using a zip library, and then stores the unpacked data appropriately in the database system (e.g., MySQL).

[1605] Output: The decompressed and decrypted data is stored in the database.

[1606] Step 4:

[1607] The server preprocesses the stored communication history and trains a natural language processing model.

[1608] Input: The deceased person's communication history stored in a database.

[1609] What it does: The server uses a natural language processing library (e.g., NLTK, spaCy) to tokenize the message text and remove unnecessary data. The preprocessed data is used to train a model to learn the unique speaking style and phrasing of the deceased.

[1610] Output: A trained natural language processing model.

[1611] Step 5:

[1612] The user uses the terminal to enter messages to interact with the system.

[1613] Input: A message typed by the user (e.g., "How was your day?")

[1614] Specific operation: The user enters a message in the input form and presses the send button.

[1615] Output: User input messages sent to the terminal.

[1616] Step 6:

[1617] The terminal sends an input message from the user to the server.

[1618] Input: User input message.

[1619] Specific operation: The terminal adds metadata such as a timestamp and user ID to the input message and sends it to the server using the HTTPS protocol.

[1620] Output: The user's input message and metadata sent to the server.

[1621] Step 7:

[1622] The server analyzes messages received from users and generates appropriate responses using a trained natural language processing model and emotion estimation engine.

[1623] Input: The message and metadata sent by the user.

[1624] How it works: The server analyzes the message content using a natural language processing library and identifies the user's sentiment using a sentiment analysis library (e.g., TextBlob, VADER). A trained generative AI model generates a response, and the sentiment estimation engine adjusts the response.

[1625] Output: The generated reply message.

[1626] Step 8:

[1627] The terminal displays the response received from the server to the user.

[1628] Input: The reply message sent by the server.

[1629] Specific operation: The terminal displays the received reply message on the screen, visually presenting it to the user.

[1630] Output: The reply message that is displayed to the user.

[1631] Step 9:

[1632] The user provides feedback on the system's response.

[1633] Input: User ratings and comments.

[1634] Specific operation: The user enters the response rating and comments in the feedback form and presses the submit button.

[1635] Output: Feedback data sent to the device.

[1636] Step 10:

[1637] The server collects feedback from users and reflects it in improving the natural language processing model and emotion estimation engine.

[1638] Input: Feedback data from users.

[1639] What it does: The server stores the feedback data in a database, analyzes it to identify areas for improvement, and retrains the model.

[1640] Output: Improved natural language processing models and sentiment estimation engines.

[1641] (Application example 2)

[1642] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1643] Conventional dialogue systems have difficulty providing a dialogue experience with the deceased, and lack emotional care when users interact with the deceased through past message history. Furthermore, there is no function to provide support for products related to the deceased within the virtual store, making it difficult for users to select products that are considerate to the deceased. Our goal is to solve these issues and provide a more personal and emotionally sensitive dialogue experience.

[1644] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1645] In this invention, the server includes means for collecting and storing basic information and communication history of the deceased, means for preprocessing the collected communication history to train a natural language processing model, means for receiving an input message from a user and generating a response using the trained natural language processing model, means for displaying the generated response to the user, means for providing support for inquiries about products related to the deceased in a virtual store, and means for adjusting the tone and content of the response using an emotion engine. This enables the user to purchase products related to the deceased in the virtual store while receiving personal support through a dialogue with the deceased.

[1646] "Deceased" refers to a person who has passed away.

[1647] "Basic information" refers to personal data such as the deceased's name and date of birth.

[1648] "Communication history" refers to past messages and conversation records between the deceased and the user.

[1649] A "natural language processing model" is an algorithm that understands and processes human language, generating responses based on input text.

[1650] "Preprocessing" refers to the process of preparing collected data in a form that is easy to analyze by tokenizing it, removing unnecessary data, performing morphological analysis, etc.

[1651] "User" refers to a person who uses this system to have an interactive experience with a deceased person.

[1652] "Input message" refers to a text message that a user sends to the system.

[1653] "Reply" refers to a response message generated by the system that mimics the speech style and emotions of the deceased.

[1654] "Display means" refers to a method or device for displaying the generated response on the user's terminal.

[1655] A "virtual store" refers to a virtual store space where products and services can be purchased online.

[1656] An "emotion engine" is an algorithm that analyzes emotions from a user's input message and adjusts the tone and content of the response based on those emotions.

[1657] The present invention is a system that allows users to have a conversational experience with a deceased person. It collects and stores basic information and communication history of the deceased, and can train a natural language processing model based on that data. Furthermore, it has the function of providing support for inquiries about products related to the deceased in a virtual store.

[1658] Overall system configuration

[1659] This system is composed of a server, a terminal, and a user, and operates as follows.

[1660] Data collection and storage

[1661] To start using the system, users input and upload basic information about the deceased person and their communication history, such as their name, date of birth, and past message history in text format. The device receives this information, compresses and encrypts the data, and sends it to the server.

[1662] The server decompresses and decrypts the received data and stores it in a database. When storing it, it properly associates the deceased person's basic information and message history with the data and stores it in secure storage, thereby protecting the user's privacy.

[1663] Data preprocessing and model training

[1664] The server preprocesses the stored message history, which includes tokenizing the messages, removing unnecessary data, and morphological analysis. The preprocessed data is then used to train a natural language processing model to learn the deceased person's unique phrasing and speaking patterns.

[1665] User interaction generation and display

[1666] The user uses the device to input a message to initiate a conversation with the system. For example, they could ask, "Did the deceased have any special memories of this accessory?" The device then sends this input message to the server, which then analyzes the message and generates an appropriate response using a trained natural language processing model. The server also uses an emotion engine to recognize emotions in the user's input message and adjust the tone and content of the response based on those emotions.

[1667] The device displays the generated response to the user, allowing the user to experience as if they were having a conversation with the deceased person.

[1668] Support functions within the virtual store

[1669] The system includes a function that provides support for inquiries about products related to the deceased in the virtual store. For example, when a user finds an accessory related to the deceased in the virtual store, they can ask about the deceased's memories and stories associated with the accessory and receive appropriate answers. This allows users to select products through a more personalized experience.

[1670] Software and hardware used

[1671] Server: Used for storing data, preprocessing, training natural language processing models, and generating responses.

[1672] Device: The device (smartphone, tablet, etc.) that a user uses to type and receive messages.

[1673] Natural language processing model: Using OpenAI GPT-3 and other models, responses are generated based on the user's input message.

[1674] Emotion engine: Analyzes the sentiment of a user's message and adjusts the tone and content of responses accordingly.

[1675] Examples of concrete examples and prompts

[1676] For example, if a user is searching for an accessory related to a deceased person in a virtual store, a possible question might be, "Did the deceased have any special memories associated with this accessory?"

[1677] Example prompt sentence:

[1678] How was your day?

[1679] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1680] Step 1:

[1681] Data collection

[1682] Users input and upload basic information and communication history of the deceased person into the device. Specifically, they provide the deceased person's name, date of birth, and past message history as a text file. The device receives this data, compresses and encrypts it, and sends it to the server.

[1683] Input: Deceased person's name, date of birth, message history.

[1684] Output: Compressed and encrypted data.

[1685] Step 2:

[1686] Data storage

[1687] The server receives the data sent from the device, decompresses and decrypts it, and then associates the deceased person's basic information and message history with a database and stores them in a secure storage environment to ensure the data remains private.

[1688] Input: Compressed and encrypted data.

[1689] Output: Basic information and message history of the deceased person stored in the database.

[1690] Step 3:

[1691] Data Preprocessing

[1692] The server preprocesses the stored message history by tokenizing the messages, removing unnecessary data, and performing morphological analysis. This process prepares the data in a format that is easy to analyze.

[1693] Input: The saved message history.

[1694] Output: Preprocessed data.

[1695] Step 4:

[1696] Training the model

[1697] The server uses the preprocessed data to train a natural language processing model, which learns the specific phrasing and speech patterns of the deceased.

[1698] Input: Preprocessed data.

[1699] Output: A trained natural language processing model.

[1700] Step 5:

[1701] Dialogue Generation

[1702] The user inputs a message to the system using the terminal, for example, a question such as "Did the deceased have any special memories about this accessory?" The terminal then sends this input message to the server.

[1703] Input: The user's input message.

[1704] Output: The input message sent to the server.

[1705] Step 6:

[1706] Response Generation

[1707] The server analyzes messages sent by users and generates appropriate responses using a trained natural language processing model, including an emotion engine to analyze the user's emotions and adjust the tone and content of the response.

[1708] Input: The input message sent to the server.

[1709] Output: The generated response.

[1710] Step 7:

[1711] Viewing the response

[1712] The terminal displays the generated response received from the server to the user, thereby allowing the user to have a conversational experience with the deceased person.

[1713] Input: The generated response.

[1714] Output: The response displayed to the user.

[1715] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1716] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1717] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1718] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1719] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1720] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1721] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1722] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1723] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1724] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1725] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1726] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1727] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1728] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1729] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1730] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1731] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1732] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1733] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1734] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1735] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1736] The following is further disclosed regarding the above embodiment.

[1737] (Claim 1)

[1738] A means of collecting and storing basic information and message history of the deceased person;

[1739] a means for preprocessing the collected message history to train a natural language processing model;

[1740] means for receiving an input message from a user and generating a response using the trained natural language processing model;

[1741] means for displaying the generated response to the user;

[1742] A system including:

[1743] (Claim 2)

[1744] 10. The system of claim 1, further comprising means for collecting feedback from users and utilizing it in training the natural language processing model to improve the model.

[1745] (Claim 3)

[1746] 10. The system of claim 1, further comprising encryption means for securely storing the collected data.

[1747] "Example 1"

[1748] (Claim 1)

[1749] A means of collecting and storing basic information and message history of the deceased person;

[1750] a means for preprocessing the collected message history to train a natural language processing model;

[1751] means for receiving an input message from a user and generating a response using the trained natural language processing model;

[1752] means for displaying the generated response to the user;

[1753] a means for adding metadata such as a timestamp to an input message from a user and transmitting the message to a server;

[1754] A system including:

[1755] (Claim 2)

[1756] 10. The system of claim 1, further comprising means for collecting feedback from users and utilizing it in training the natural language processing model to improve the model.

[1757] (Claim 3)

[1758] 10. The system of claim 1, further comprising encryption means for securely storing the collected data.

[1759] (Claim 4)

[1760] 10. The system of claim 1, further comprising means for uploading basic information and message history of the deceased person to the terminal as the user provides it.

[1761] (Claim 5)

[1762] 10. The system of claim 1, further comprising means for filtering out and tokenizing the collected message history to create a data set.

[1763] (Claim 6)

[1764] 10. The system of claim 1, further comprising means for using a trained natural language processing model to learn the deceased person's unique speaking style and phrasing.

[1765] (Claim 7)

[1766] 10. The system of claim 1, further comprising means for generating a response that the deceased would likely make based on a message entered by the user.

[1767] "Application Example 1"

[1768] (Claim 1)

[1769] A means of collecting and storing basic information and message history of the deceased person;

[1770] a means for preprocessing the collected message history to train a natural language processing model;

[1771] means for receiving an input message from a user and generating a response using the trained natural language processing model;

[1772] means for displaying the generated response to the user;

[1773] A system that includes an application program installed on a terminal at a physical store.

[1774] (Claim 2)

[1775] 10. The system of claim 1, further comprising means for collecting feedback from users and utilizing it in training the natural language processing model to improve the model.

[1776] (Claim 3)

[1777] 10. The system of claim 1, further comprising encryption means for securely storing the collected data.

[1778] "Example 2: Combining Emotion Engines"

[1779] (Claim 1)

[1780] A means for users to input and upload basic information about the deceased person and their communication history;

[1781] means for securely processing and transmitting entered and uploaded data to a server;

[1782] means for decompressing and decrypting the received data and storing it in a database;

[1783] a means for preprocessing the stored communication history, tokenizing and parsing the data, and training a natural language processing model;

[1784] means for receiving an input message from a user and generating a reply using a trained natural language processing model and an emotion estimation engine;

[1785] means for displaying the generated response to the user;

[1786] A system including:

[1787] (Claim 2)

[1788] 10. The system of claim 1, further comprising means for collecting feedback from users and utilizing it as training data for the natural language processing model to retrain and improve the model.

[1789] (Claim 3)

[1790] 10. The system of claim 1, further comprising encryption means for increasing security during storage and transmission of collected data.

[1791] "Application example 2 when combining emotion engines"

[1792] (Claim 1)

[1793] A means of collecting and storing basic information and communication history of the deceased;

[1794] a means for preprocessing the collected communication history to train a natural language processing model;

[1795] means for receiving an input message from a user and generating a response using the trained natural language processing model;

[1796] means for displaying the generated response to the user;

[1797] a means for providing support for inquiries regarding products related to the deceased person within the virtual store;

[1798] A means to adjust the tone and content of responses using an emotion engine;

[1799] A system including:

[1800] (Claim 2)

[1801] 10. The system of claim 1, further comprising means for collecting feedback from users and utilizing it in training the natural language processing model to improve the model.

[1802] (Claim 3)

[1803] 10. The system of claim 1, further comprising encryption means for securely storing the collected data. [Explanation of symbols]

[1804] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting and storing basic information and message history of the deceased person; a means for preprocessing the collected message history to train a natural language processing model; means for receiving an input message from a user and generating a response using the trained natural language processing model; means for displaying the generated response to the user; A system including:

2. The system of claim 1 , further comprising means for collecting feedback from users and utilizing it in training the natural language processing model to improve the model.

3. The system of claim 1 further comprising encryption means for securely storing collected data.

Images