system

Abstract

Provide a system. The solution involves collecting data such as emails, photos, videos, voice messages, social network service logs, and letters from the electronic devices of a deceased family member; a means for converting the collected data into a text format and preprocessing the data; A means of training a generative AI model based on the preprocessed data; and A means to build a chatbot using a trained generative AI model; A means of integrating the constructed chatbot into a communication interface; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, there are limited means for bereaved family members to relive personal memories and records of their deceased loved ones. This leaves few opportunities for them to receive emotional support and makes it difficult for them to experience a sense of dialogue with their deceased loved ones. Furthermore, due to a lack of technology to efficiently utilize acquired digital data in a single, integrated form, organizing and analyzing the data left behind is complex and time-consuming. The present invention aims to solve these problems and provide a system that allows bereaved family members to more easily and naturally experience a dialogue with their deceased loved ones. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes: means for collecting data, such as emails, photos, videos, voice messages, social networking service logs, and letters, from the electronic devices of deceased family members; means for converting the collected data into text format and preprocessing it; means for training a generative AI model based on the preprocessed data; means for building a chatbot using the trained generative AI model; and means for integrating the built chatbot into a communication interface. In particular, by performing optical character recognition (OCR) on the preprocessed data to convert image and video data into text, and by performing voice recognition to convert voice data into text, data integration and analysis can be efficiently performed. As a result, bereaved family members can recreate the conversational experience with their deceased family members and provide emotional support.

[0006] "Electronic devices" are devices for storing and processing data such as emails, photos, videos, voice messages, social networking service logs, and letters.

[0007] "Data collection" is the process of obtaining necessary information from the electronic devices of deceased family members.

[0008] "Text format" refers to a character-based data format, which is digital information converted into a form that can be processed by a computer.

[0009] "Preprocessing" refers to a series of processes carried out to prepare collected data in a form that is easy to analyze.

[0010] Optical character recognition (OCR) is a technology that digitizes characters from paper documents and image files and converts them into text information.

[0011] "Speech recognition" is a technology that converts voice data into text data.

[0012] A "generative AI model" is a trained algorithm that uses artificial intelligence to generate responses in natural language, just like a human.

[0013] "Training" is the process of optimizing the performance of an AI model using collected data.

[0014] A "chatbot" is a virtual assistant programmed to simulate human interaction.

[0015] A "communications interface" is an interactive interface for exchanging data between different computer systems or networks. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention relates to a system that learns the digital information of deceased family members and recreates it as a chatbot that can be interacted with 24 hours a day via a LINE account. This system is composed of elements that play the roles of a server, a terminal, and a user.

[0038] System Configuration

[0039] Server-side processing

[0040] 1. Data Collection

[0041] The server collects data from the deceased family member's electronic devices, including emails, photos, videos, voice messages, social networking logs, and letters. This data is organized into the deceased family member's digital legacy.

[0042] 2. Data Preprocessing

[0043] The collected data is preprocessed on the server. Specifically, text data is normalized, text is extracted from images and videos using optical character recognition (OCR), and voice messages are converted to text using speech recognition. This unifies the data from various formats into a unified text format.

[0044] 3. Model Training

[0045] The pre-processed data is sent to a server as a training dataset for a generative AI model, which learns from this dataset and is tuned to generate responses that mimic the language and thought patterns of the deceased family member.

[0046] 4. Chatbot construction

[0047] Using the trained generative AI model, the server builds a chatbot that responds appropriately to messages from users.

[0048] 5. LINE account integration

[0049] The chatbot is integrated into a LINE account through a communication interface. The server uses the LINE Messaging API to manage communication between the chatbot and the LINE platform.

[0050] User processing

[0051] When a user sends a message to the chatbot for their deceased family member through their LINE account, the message is sent to the server, where the generative AI model analyzes the message and generates an appropriate response, which is then sent back to the user via the LINE API.

[0052] Specific use cases

[0053] For example, consider the case where a user sends a message to a chatbot via their LINE account saying, "Show me some recent photos of my family."

[0054] 1. A user sends a message through their LINE account.

[0055] Message: "Show me a recent photo of your family."

[0056] 2. The device sends the message to the LINE server.

[0057] 3. The LINE server passes the message to the chatbot on the server.

[0058] 4. The server's generative AI model analyzes the user's message and generates an appropriate response.

[0059] Example response: "Here's a recent photo of my family." with a link or image file showing the photo

[0060] 5. The server sends the generated response through the LINE server.

[0061] 6. The user receives a response from the server in their LINE account and can view recent photos of their family.

[0062] This allows users to recreate conversations with their deceased family members, providing an experience that maintains an emotional connection. The system provides a way for bereaved families to receive emotional support while keeping the memory of their deceased family members a part of their lives.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] The server collects data from the deceased family member's electronic devices, including emails, photos, videos, voice messages, social networking service logs, and letters. Specifically, it uses the API provided by the device to download the data and convert it into an appropriate data format. This allows various data to be aggregated in one place.

[0066] Step 2:

[0067] The server pre-processes the collected data, which includes the following steps:

[0068] Normalization of text data (emails, SNS logs), specifically removing punctuation, adjusting spaces, expanding slang and abbreviations, etc.

[0069] Perform optical character recognition (OCR) on image and video data to extract text.

[0070] Voice messages are converted into text using voice recognition technology.

[0071] Step 3:

[0072] The server trains a generative AI model based on the preprocessed text data. Specifically, the collected text data is tokenized and provided as input to the generative AI model. The model is tuned to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[0073] Step 4:

[0074] The server uses the trained generative AI model to build a chatbot. The built chatbot has the ability to receive input from the user and generate an appropriate response. The actual response that the chatbot generates depends on what the AI ​​model has learned.

[0075] Step 5:

[0076] The server integrates the built chatbot into the communication interface. Specifically, it uses the LINE Messaging API to link the chatbot to a specific LINE account. This integration allows users to converse with the chatbot through their LINE account.

[0077] Step 6:

[0078] The user sends a message to the chatbot through their LINE account.

[0079] For example, if a user sends a message saying, "Show me the last photo of your family you took," the message is immediately transmitted to the LINE server.

[0080] Step 7:

[0081] The LINE server forwards the received message to the server, which then receives it and has the generative AI model analyze the message.

[0082] Step 8:

[0083] The server's generative AI model analyzes the received message and generates an appropriate response, such as "Here's the last photo of the family we took," along with a link to the photo.

[0084] Step 9:

[0085] The server sends the generated response back to the LINE server, which then sends it to the user's LINE account. The user can then view photos of the specified family member.

[0086] Through this series of steps, users can recreate the interaction experience with their deceased family members and maintain an emotional connection. The system provides a way for bereaved families to receive emotional support while keeping records of their deceased family members a part of their lives.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] In modern society, there is a lack of ways to continue to recreate memories and emotional connections with deceased family members. Existing technologies have difficulty integrating the diverse digital information of deceased family members to recreate a consistent interactive experience. Therefore, there is a need for a system that allows bereaved family members to receive emotional support while continuing to cherish memories of their deceased family members.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes means for collecting information such as emails, image data, video data, audio data, social network service logs, and document data from the information devices of the deceased family member, means for converting the collected data into text format and preprocessing it, means for training an artificial intelligence model generated based on the preprocessed data, means for constructing a dialogue agent using the trained generative artificial intelligence model, means for integrating the constructed dialogue agent into a communication interface, means for generating information based on the collected data and linking the dialogue agent to an external system, and means for analyzing the information and generating responses to inquiries from users. This makes it possible to recreate an emotional connection through natural dialogue using the digital information of the deceased family member, allowing the bereaved family to receive emotional support.

[0092] An "information device" is a device that has the function of generating, saving, and transferring e-mail, image data, video data, audio data, social network service logs, document data, and the like.

[0093] A "data collection means" is a function or process for collecting specific digital data from information devices and transferring it to a server.

[0094] "Preprocessing means" refers to a function or process that analyzes collected digital data and converts the data into a text format to unify the format.

[0095] A "generated artificial intelligence model" is an AI technology that has the ability to learn from a specific dataset and generate responses in natural language.

[0096] A "dialogue agent" is a program or system that has the ability to respond to messages from a user using a generated artificial intelligence model.

[0097] A "communication interface" is a protocol or mechanism for sending and receiving information between an interactive agent and an external messaging platform.

[0098] "External system integration means" is a function that enables a conversational agent to operate in conjunction with other systems or platforms (e.g., messaging apps).

[0099] The "information analysis means" is a function or process for analyzing messages from users and generating optimal response content.

[0100] The "response generation means" is a function or process for generating an appropriate response to be provided to the user based on the analysis results.

[0101] This invention relates to a system that trains a generative AI model based on the digital information of deceased family members and recreates it as a chatbot that can interact with them 24 hours a day. Specific embodiments of this system are described below.

[0102] Data collection

[0103] The server collects various information from the information devices of deceased family members, including emails, image data, video data, audio data, social network service logs, and document data. Dedicated scripts and programs are used in the collection process, and this data is transferred to the server. Specifically, this occurs when a user uploads a photo folder on their smartphone using a dedicated application.

[0104] Data Preprocessing

[0105] The server analyzes the collected data and converts it into a text format to standardize it. It uses optical character recognition (OCR) technology, such as Google® Cloud Vision API or Tesseract, to normalize the text data and process image and video data. It also converts voice messages into text using speech recognition technology, such as Google Speech-to-Text API. This ensures that all data is in a standardized text format.

[0106] Model learning

[0107] The server uses the preprocessed data to train a generative AI model, specifically OpenAI's GPT-3 or ChatGPT models. These models are trained to generate natural responses based on the deceased family member's vocabulary and thought patterns based on the text data.

[0108] Chatbot construction

[0109] The server uses the trained generative AI model to build a conversational agent, which is implemented as an API using the Python Flask framework, resulting in a chatbot that responds to messages from users.

[0110] LINE account integration

[0111] The server integrates the built conversational agent with a LINE account. Specifically, it uses the LINE Messaging API to manage communication between the chatbot and the LINE platform. When a user sends a message via LINE, the message is forwarded to the server, where the generative AI model analyzes it and generates an appropriate response. The generated response is then sent back to the user via the LINE API.

[0112] Examples of concrete examples and prompts

[0113] For example, here is a specific example of a user sending a request to a chatbot via their LINE account saying, "Show me recent photos of my family."

[0114] 1. A user sends a message through their LINE account.

[0115] Message: "Show me a recent photo of your family."

[0116] 2. The server's AI model analyzes this message and generates an appropriate response.

[0117] Example response: "Here's a recent photo of my family." with a link or image file showing the photo

[0118] Thus, a specific example of a prompt that a generative AI model can use to generate an appropriate response is "Hello, can you show me a recent family photo?" Because the model is trained on this prompt, it can provide a natural response to the user.

[0119] In this way, the bereaved can relive the conversations with their deceased loved ones and maintain emotional ties, which is a specific embodiment of the present invention.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1:

[0122] Data collection

[0123] Input: Emails, image data, video data, audio data, social networking service logs, document data, etc. from the information devices of deceased family members.

[0124] Output: The collection of digital data transferred to the server.

[0125] Specific operation: The user uploads data from their smartphone or PC using a dedicated application. The data is collected by a dedicated script and transferred to the server. For example, the user selects a photo folder on their smartphone and presses the upload button, which sends the data to the server.

[0126] Step 2:

[0127] Data Preprocessing

[0128] Input: A collection of digital data transferred to the server.

[0129] Output: A normalized set of data converted to text format.

[0130] What it does: The server analyzes the collected data and converts it into text format. For example, it uses the Google Cloud Vision API to extract text from images and video data, and the Google Speech-to-Text API to convert audio data into text. Furthermore, it performs spell checking and grammar correction as part of the normalization process.

[0131] Step 3:

[0132] Model learning

[0133] Input: Preprocessed data in text format.

[0134] Output: A trained generative AI model.

[0135] How it works: The server inputs the preprocessed data into a generative AI model (e.g., GPT-3) and trains the model. Based on a large amount of text data, the model learns the language and thought patterns of the deceased family member. A dedicated training script is used for this training process. For example, preprocessed text data from emails or letters is input into the model, and training proceeds over a period of several hours to several days.

[0136] Step 4:

[0137] Chatbot construction

[0138] Input: A trained generative AI model.

[0139] Output: A chatbot API that can respond to user input.

[0140] How it works: The server uses a trained generative AI model to build a conversational agent. An API server is configured using Python's Flask framework, and the generative AI model is connected to this API. This creates a mechanism for the generative AI model to respond appropriately to external requests. For example, if an endpoint called " / message" is set up on the Flask server, when a message is POSTed there, the generative AI model will generate and return a response.

[0141] Step 5:

[0142] LINE account integration

[0143] Input: A LINE message from the user.

[0144] Output: The response of the generated AI model through a LINE account.

[0145] How it works: The server integrates the conversational agent into a LINE account using the LINE Messaging API. Create a channel on the LINE Developers site and set the API key in the Flask server. When a user sends a message from the LINE app, the message is forwarded to the server, where the generative AI model analyzes it and generates a response, which is then sent back to the user via the LINE API. For example, when a user sends "Hello" on LINE, the server passes the message to the generative AI model, which generates the response "Hello, how are you?" and sends it back to the LINE app.

[0146] Step 6:

[0147] User interaction

[0148] Input: A message from the user.

[0149] Output: The response message generated by the generative AI model.

[0150] Specific operation: A user sends a message to their LINE account. The device sends the message to the LINE server, which passes it on to the server's chatbot. The generative AI model analyzes the message, generates an appropriate response, and sends it back to the user via the LINE API. The user receives the response in their LINE account and can continue the conversation. For example, if a user sends "Show me some recent photos of your family," the server retrieves the latest photo from the database and sends it along with the response "Here are some recent photos of your family."

[0151] (Application example 1)

[0152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] In recent years, bereaved families have increasingly sought emotional support and psychological healing, but existing technologies have limited the means to provide this digitally. Furthermore, with the spread of autonomous vehicles, how users spend their time in their cars has become an important issue. In response to this, there is a need to provide a safe means for them to communicate with their deceased family members.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0155] In this invention, the server includes: means for collecting data such as emails, images, videos, voice messages, social network logs, and letters from the electronic devices of the deceased family member; means for converting the collected data into text format and preprocessing it; means for training a generative AI model based on the preprocessed data; means for building a chatbot using the trained generative AI model; means for integrating the built chatbot into a communication interface; and means, mounted on the smart glasses, for recognizing voice input and sending user dialogue commands to the chatbot. This allows bereaved family members to experience an emotional connection with their deceased family member through dialogue with them in an autonomous vehicle and safely find healing.

[0156] "Electronic device" is a general term for devices that generate, store, transmit, or receive digital data.

[0157] "Data" refers to information such as emails, images, videos, voice messages, social network logs, letters, etc.

[0158] "Means of collection" refers to the process for extracting specific data from electronic devices and transferring it to a server, etc.

[0159] "Preprocessing means" refers to the process of converting collected data into a unified text format and organizing it in a way that makes it easier to analyze.

[0160] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses to user input based on data.

[0161] "Training" refers to the process of using preprocessed data to train a generative AI model to learn specific patterns.

[0162] A "chatbot" is a program that can understand natural language and respond to users through dialogue.

[0163] "Means of integration into a communication interface" means connecting the chatbot to a specific communication platform (e.g., LINE) and enabling it to send and receive messages.

[0164] "Smart glasses" are wearable devices that have built-in displays and sensors, display information to the user, and can be operated by voice input, touch input, etc.

[0165] "Means for recognizing voice input and transmitting user interaction commands to the chatbot" refers to the process of using a microphone and software built into the smart glasses to convert voice input into text and transmit that text to the chatbot.

[0166] An "autonomous vehicle" is a vehicle that uses sensors and AI to drive autonomously without the need for driver operation.

[0167] The system for implementing this invention aims to safely recreate conversations with deceased family members. Specifically, it collects data from the deceased family member's electronic devices, builds a chatbot using a generative AI model based on that data, and enables the user to interact with the chatbot through smart glasses.

[0168] Server-side processing

[0169] 1. Data Collection

[0170] The server collects the digital information of deceased family members (e.g., emails, images, videos, audio messages, social network logs, letters, etc.), allowing for a complete picture of their digital lives.

[0171] 2. Data Preprocessing

[0172] The collected data is preprocessed on the server. Specifically, text data is normalized, and text is extracted from images and videos using optical character recognition (OCR). Voice messages are then converted to text using speech recognition. This preprocessing process converts data from a variety of formats into a unified text format.

[0173] 3. Model Training

[0174] The preprocessed data is sent to a server as a training dataset for a generative AI model (e.g., GPT-3, BERT). The generative AI model learns from this dataset and is tuned to generate responses that mimic the language and thought patterns of the deceased family member.

[0175] 4. Chatbot construction

[0176] Using the trained generative AI model, the server builds a chatbot that can respond appropriately to messages from users.

[0177] 5. Communication interface integration

[0178] The constructed chatbot is integrated into a communication interface (e.g., LINE API), through which the server manages communication between the chatbot and the user.

[0179] User processing

[0180] 1. Voice input

[0181] Suppose a user is riding in a self-driving vehicle and wearing smart glasses. When the user speaks, "Mom, what's the weather like today?", the smart glasses recognize this as voice input. The smart glasses have a built-in microphone that captures this voice input.

[0182] 2. Voice Recognition

[0183] Software installed on the smart glasses (e.g., voice recognition software using Google Cloud API) converts this voice input into text, allowing the user's commands to be captured in text format.

[0184] 3. Send a message

[0185] The captured text is sent from the smart glasses via a communication interface to a server, where a generative AI model analyzes the user's message and generates an appropriate response.

[0186] 4. Receiving a Response

[0187] The generated response is then sent back to the smart glasses via the communication interface and presented to the user in voice or text format, for example, "The weather is sunny today, it looks like it's going to be a nice day."

[0188] Specific examples

[0189] As a concrete example, consider the case where a user in a self-driving vehicle asks, "Mom, how's the weather today?" The smart glasses recognize the speech, convert it into text, and then send it to the server. The server's generative AI model analyzes the message and generates a response to the user: "The weather is sunny today. It looks like it's going to be a nice day."

[0190] Prompt Sentence Examples

[0191] Speech prompt: "Mom, how's the weather today?"

[0192] Chatbot prompt: "A 24-hour conversational chatbot that responds based on digital information about deceased family members."

[0193] This allows users to safely interact with deceased family members from within their self-driving vehicles and receive emotional support.

[0194] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0195] Step 1:

[0196] The server collects data from the deceased family member's electronic devices, including emails, images, videos, audio messages, social network logs, and letters. The data is transferred using APIs and file transfer protocols. The input is various digital data, and the output is a unified data set stored on the server.

[0197] Step 2:

[0198] The server preprocesses the collected data by normalizing the text data, extracting text from images and videos using optical character recognition (OCR), and converting voice messages into text using speech recognition software. The input is the collected data, and the output is normalized text data.

[0199] Step 3:

[0200] The server trains a generative AI model based on the preprocessed data. Using the preprocessed data as a training dataset, an AI algorithm (e.g., GPT-3, BERT) learns language and thought patterns. The input is normalized text data, and the output is a trained generative AI model.

[0201] Step 4:

[0202] The server builds a chatbot using a trained generative AI model, which enables the AI ​​model to generate natural and appropriate responses to user input. The input is the trained generative AI model, and the output is the chatbot.

[0203] Step 5:

[0204] The server integrates the constructed chatbot into a communication interface (e.g., LINE API), allowing users to interact with the chatbot through their LINE accounts. The input is connection information between the chatbot and the communication interface, and the output is the established communication state.

[0205] Step 6:

[0206] A user gets into an autonomous vehicle and puts on smart glasses. The user speaks to the vehicle, saying, "Mom, what's the weather like today?" The input is the user's voice command, and the output is the audio data of the voice command.

[0207] Step 7:

[0208] The smart glasses use a built-in microphone to recognize the user's voice and capture acoustic data, then use speech recognition software to convert the speech into text, with the input being acoustic data and the output being text data.

[0209] Step 8:

[0210] The smart glasses send the acquired text data to the server through the communication interface. The input is the text data and the output is the transmitted text message.

[0211] Step 9:

[0212] The server uses a generative AI model to analyze the received text message and generate an appropriate response. The input is the text message from the user, and the output is the generated response text.

[0213] Step 10:

[0214] The server sends the generated response to the smart glasses through the communication interface, where the input is the generated response text and the output is the text or audio displayed on the smart glasses.

[0215] Step 11:

[0216] The smart glasses provide the received response to the user in voice or text format. The input is the transmitted response text, and the output is the information presented to the user. Specifically, the response may be something like, "The weather is sunny today, it looks like it's going to be a nice day."

[0217] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0218] This invention combines an emotion engine with a system that learns the digital information of deceased family members and recreates them as chatbots that can be interacted with 24 hours a day via LINE accounts. The system is composed of elements that play the roles of a server, a device, and a user.

[0219] System Configuration

[0220] Server-side processing

[0221] 1. Data Collection

[0222] The server collects data such as emails, photos, videos, voice messages, social networking service logs, and letters from the deceased family members' electronic devices. This involves using the device's API to obtain the data and transferring it to the server. The collected data is then stored in a database and managed centrally.

[0223] 2. Data Preprocessing

[0224] The server preprocesses the collected data. Specifically, it normalizes text data (e.g., emails, social media logs) and removes punctuation and special characters. It also performs optical character recognition (OCR) on images and videos to extract text data. Voice messages are converted to text using speech recognition. This ensures that all data is converted into a unified text format.

[0225] 3. Model Training

[0226] The server trains a generative AI model based on the preprocessed text data. Specifically, the text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[0227] 4. Chatbot construction

[0228] The server uses a trained generative AI model to build a chatbot. The chatbot receives input from the user and generates an appropriate response. The server also combines an emotion engine to recognize the user's emotions and provide a response according to the user's emotions.

[0229] 5. LINE account integration

[0230] The server integrates the chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. This integration allows users to converse with the chatbot through their LINE account.

[0231] User processing

[0232] When a user sends a message to the chatbot for a deceased family member through their LINE account, the message is sent to the server. The server's generative AI model analyzes the message and generates an appropriate response. The server's emotion engine recognizes the user's emotion from the message content and generates a response based on that emotion. The generated response is then sent back to the user via the LINE API.

[0233] Specific use cases

[0234] For example, if a user sends "I'm tired today" to the chatbot through their LINE account, the following will happen:

[0235] 1. A user sends a message through their LINE account.

[0236] Message: "I'm tired today."

[0237] 2. The device sends the message to the LINE server.

[0238] 3. The LINE server passes the message to the chatbot on the server.

[0239] 4. The server's generative AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the emotion "tired" and generates a response that includes words of comfort for fatigue.

[0240] Sample response: "Good work. Have a good rest today."

[0241] 5. The server generates a response and sends it back to the LINE server, which then sends it to the user's LINE account.

[0242] 6. The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[0243] In this way, users can recreate the interaction experience with their deceased family members and, with the help of the emotion engine, receive more advanced emotional support, providing a means for bereaved families to receive better emotional support and cherish the memories of their deceased family members.

[0244] The processing flow will be explained below.

[0245] Step 1:

[0246] The server collects data such as emails, photos, videos, voice messages, social networking service logs, and letters from the electronic devices of deceased family members. Specifically, it retrieves and aggregates various digital data stored on the electronic devices using APIs and file transfer protocols.

[0247] Step 2:

[0248] The server pre-processes the collected data, which includes:

[0249] Normalize text data (e.g., emails and social media logs) to remove punctuation and special characters.

[0250] Optical character recognition (OCR) is used on image and video data to extract text information.

[0251] Voice messages are converted into text using voice recognition technology, which ensures that all data is converted into a unified text format.

[0252] Step 3:

[0253] The server trains a generative AI model based on the preprocessed text data, using a large dataset to train the model to reflect the deceased family member's vocabulary and thought patterns, enabling the AI ​​model to generate appropriate natural language responses.

[0254] Step 4:

[0255] The server uses the trained generative AI model to build a chatbot, which receives messages from users and generates appropriate responses.

[0256] Step 5:

[0257] The server recognizes the user's emotions using an emotion engine. Specifically, the emotion engine analyzes the input message and identifies the user's emotions (e.g., joy, sadness, anger, fatigue, etc.) using an emotion analysis algorithm.

[0258] Step 6:

[0259] Based on the analysis results of the emotion engine, the server uses a generative AI model to generate a response based on the emotion. This response is tailored to the user's current emotional state. For example, if a user sends a message saying, "I'm tired today," the emotion engine will recognize the emotion "fatigue," and the generative AI model will generate a comforting message such as, "Thank you for your hard work. Take a good rest today."

[0260] Step 7:

[0261] The server integrates the built chatbot into the communication interface. Specifically, it uses the LINE Messaging API to link the chatbot to a LINE account. This integration allows users to converse with the chatbot through their LINE account.

[0262] Step 8:

[0263] The user sends a message to the chatbot of their deceased family member through their LINE account. At this time, the user's message is sent to the server via the LINE server.

[0264] Step 9:

[0265] The server analyzes the message received from the LINE server using the generative AI model and generates an appropriate response. This includes the process where the emotion engine recognizes the user's emotions and generates a response that corresponds to those emotions.

[0266] Step 10:

[0267] The server sends the generated response back to the LINE server, which then sends it to the user's LINE account.

[0268] Step 11:

[0269] The user receives a response from the server on their LINE account and checks its contents. For example, they may receive a message such as "Thank you for your hard work. Please take a good rest today," providing emotional support.

[0270] This series of steps allows users to re-create conversations with deceased family members and, with the help of an emotion engine, receive more personalized emotional support.

[0271] Example 2

[0272] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0273] With limited means to store and relive memories of deceased family members, bereaved families find it difficult to receive emotional support or comfort through interactive experiences. To solve this problem, an advanced dialogue system that utilizes the digital information of deceased family members and also provides emotional support is needed.

[0274] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0275] In this invention, the server includes means for collecting data such as emails, images, videos, voice communications, social network service records, and letters from the information devices of the deceased family member, means for converting the collected data into text format and performing preprocessing, means for training a generative AI model based on the preprocessed data, means for constructing a dialogue system using the trained generative AI model, means for integrating the constructed dialogue system into a communication interface, emotion engine means for analyzing messages from users and recognizing their emotions, and means for generating responses based on the recognition results of the emotion engine. This allows the bereaved family to relive conversations with their deceased family member and receive support tailored to their emotions.

[0276] "Information equipment" is a general term for devices that can generate, store, manage, and transmit data such as emails, images, videos, voice communications, social network service records, and letters.

[0277] "Data collection means" refers to the functions and processes that automatically or manually acquire data from various information devices and store it on a central server.

[0278] "Data preprocessing measures" are processes such as normalization, removal of punctuation and special characters, optical character recognition (OCR) and speech recognition to convert collected data into a unified text format.

[0279] A "generative AI model" is an algorithm or artificial intelligence (AI) system that is trained based on collected text data to understand specific patterns and contexts and generate appropriate responses.

[0280] A "dialogue system" is a software application built using generative AI models to engage in dialogue with users and provide appropriate responses to their questions and comments.

[0281] A "communication interface" is a mechanism for connecting a dialogue system with a user, and often uses messaging platforms such as LINE or email.

[0282] "Emotion engine means" refers to algorithms and technologies for analyzing messages from users and recognizing the emotions contained in the content.

[0283] "Response generation means" refers to a process or function for generating an appropriate response message based on the emotion recognized by the emotion engine.

[0284] "Preprocessed data" refers to raw data collected that has been converted into text format and processed to a state suitable for normalization and analysis.

[0285] This invention relates to a system for building chatbots that can converse in real time based on the digital information of deceased family members. This system includes elements of a server, a terminal, and a user, and by combining an emotion engine, provides more advanced emotional support.

[0286] First, the server collects data from the deceased family member's electronic devices. Specifically, the hardware used utilizes data collection APIs (e.g., Gmail API, Google Photos API) to obtain data such as emails, photos, videos, voice messages, social networking service records, and letters. This data is then stored in a database on the server and managed centrally.

[0287] Next, the server preprocesses the collected data. The collected data is not suitable for analysis or training generative AI models as it is, so it must be converted into text format. Text data is normalized and punctuation and special characters are removed, and image and video data undergoes optical character recognition (OCR). Audio data is then converted into text using speech recognition technology (e.g., Google Speech-to-Text API). This preprocessing ensures that all data is converted into a unified text format.

[0288] The server then trains a generative AI model using the preprocessed text data. The generative AI model uses a natural language processing library (e.g., Transformers). This process involves providing the preprocessed data as input to the AI ​​model, training it to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[0289] Once the learning is complete, the server uses the trained generative AI model to build a chatbot. The chatbot receives messages from users and generates appropriate responses. Furthermore, by incorporating an emotion engine (e.g., Sentiment Analysis) to recognize the user's emotions, the chatbot provides responses that correspond to the user's emotions.

[0290] The server then integrates the chatbot into the communication interface. This process involves linking the chatbot to a LINE account using the LINE Messaging API. This integration allows users to interact with the chatbot through the LINE platform.

[0291] When a user sends a message through their LINE account, the message is sent via their device to the LINE server, where it is then forwarded to a chatbot on the server. The generative AI model on the server analyzes the user's message and generates an appropriate response. The emotion engine recognizes the user's emotion from the content of the message and generates a response based on that emotion. The generated response is then sent back to the user via the LINE API.

[0292] Specific examples

[0293] For example, if a user sends "I'm tired today" to a chatbot through their LINE account, the process will be as follows:

[0294] 1. A user sends a message saying, "I'm tired today."

[0295] 2. The device sends the message to the LINE server.

[0296] 3. The LINE server forwards the message to the chatbot on the server.

[0297] 4. The server's generative AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the emotion "tired" and generates a response that includes words of comfort for fatigue.

[0298] Example response: "Good work. Have a good rest today."

[0299] 5. The server generates a response and sends it back to the LINE server.

[0300] 6. The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[0301] In this way, users can recreate the interaction experience with their deceased family members and, with the help of the emotion engine, receive higher levels of emotional support.

[0302] Prompt Sentence Examples

[0303] Here are some examples of input prompts for generative AI models:

[0304] "I've been so busy and tired today. Could you please give me some words of encouragement?"

[0305] "I've been feeling a bit down lately...what should I do?"

[0306] "I'm really looking forward to talking to you."

[0307] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0308] Step 1:

[0309] Data collection

[0310] The server collects data such as emails, images, videos, voice communications, social networking service records, and letters from the information devices of deceased family members.

[0311] Specifically, it uses the Gmail API to collect emails, the Google Photos API to download photos and videos, and retrieves voice messages from the device's storage.

[0312] Input: Data retrieval request from information device.

[0313] Output: Raw data (emails, images, videos, audio messages, etc.) stored in a server database.

[0314] Step 2:

[0315] Data Preprocessing

[0316] The server preprocesses the collected raw data.

[0317] Text data is normalized to remove punctuation and special characters, image and video data is converted to text using OCR (Optical Character Recognition), and voice messages are converted to text using speech recognition technology.

[0318] As a concrete example, we use the Google Speech-to-Text API to convert voice messages into text and Tesseract OCR to extract text from images.

[0319] Input: Raw data stored in the server database.

[0320] Output: Preprocessed text data.

[0321] Step 3:

[0322] Model learning

[0323] The server uses the preprocessed text data to train a generative AI model.

[0324] Use a natural language processing library (e.g., Transformers) to train your AI model using preprocessed data.

[0325] Specifically, the collected data sets are batch processed and input into the AI ​​model sequentially.

[0326] Input: Preprocessed text data.

[0327] Output: A trained generative AI model.

[0328] Step 4:

[0329] Chatbot construction

[0330] The server builds a chatbot using a trained generative AI model.

[0331] Chatbots have the ability to analyze messages from users and generate appropriate responses. Furthermore, by incorporating an emotion engine (e.g., Sentiment Analysis), they can recognize the user's emotions and provide responses that correspond to those emotions.

[0332] Specifically, it incorporates a sentiment analysis library to add emotional elements to the generated responses.

[0333] Input: Trained generative AI model and emotion engine.

[0334] Output: Available chatbot systems.

[0335] Step 5:

[0336] LINE account integration

[0337] The server integrates the built chatbot into the communication interface.

[0338] Use the LINE Messaging API to link your chatbot to your LINE account, allowing users to interact with your chatbot through the LINE Platform.

[0339] Specifically, the API endpoint is set using the channel access token obtained from the LINE Developer Console and registered in the chatbot's message receiving mechanism.

[0340] Input: Chatbot system and LINE Messaging API.

[0341] Output: Chatbot linked to LINE account.

[0342] Step 6:

[0343] Sending a message and generating a response

[0344] When a user sends a message through their LINE account, the message is sent via the device to the LINE server, and then forwarded to a chatbot on the server.

[0345] The server's generative AI model analyzes the user's message and generates an appropriate response. The emotion engine recognizes the user's emotion from the message content and generates a response based on that emotion.

[0346] Example: When a user sends "I'm tired today," the emotion engine recognizes the emotion "tired" and generates a response like "Great work. Take a good rest today."

[0347] Input: A message from the user.

[0348] Output: An appropriate response message that takes emotion into consideration.

[0349] Step 7:

[0350] Returning a response

[0351] The server generates a response message and sends it back to the LINE server, which then sends it to the user's LINE account.

[0352] The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[0353] Specifically, the generated response message is sent as an API request to the LINE Messaging API and delivered to the user's LINE account.

[0354] Input: The server-generated response message.

[0355] Output: A response message that will be displayed on the user's LINE account.

[0356] (Application example 2)

[0357] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0358] In today's world, the means for remembering the deceased are limited, making it difficult to maintain emotional connections. Furthermore, the means for identifying and purchasing products and mementos related to the deceased are not highly personalized. As a result, it is difficult for bereaved families to experience a memorial experience that is emotionally supported. This invention aims to solve this problem by providing a system that learns the digital information of the deceased, provides emotional support through dialogue with the user, and recommends products related to the deceased.

[0359] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0360] In this invention, the server includes: means for collecting data such as electronic messages, electronic images, electronic videos, audio messages, social network service logs, and letters from the electronic devices of the deceased family member; means for converting the collected data into text format and preprocessing it; means for training a generative AI model based on the preprocessed data; means for building a chatbot using the trained generative AI model; means for integrating the built chatbot into a communication interface; and means for providing personalized product recommendations through the user interface. This allows the user to maintain an emotional connection with the deceased through conversations with the deceased, receive recommendations for products related to the deceased, and enjoy a special shopping experience.

[0361] An "electronic message" is text information sent or received in digital form, such as an email or social media message.

[0362] "Electronic image" refers to a still image or photograph stored in digital format.

[0363] "Electronic moving images" refers to moving images or video clips stored in digital format.

[0364] A "voice message" is a message that includes recorded voice data.

[0365] "Social Network Service Logs" refers to records of user activities and interactions on social networking services.

[0366] A "letter" refers to a written communication, either on paper or digitally.

[0367] A "text format" is a format that describes information using letters and symbols.

[0368] "Preprocessing" refers to the process of converting and organizing data into a form suitable for further processing and analysis.

[0369] A "generative AI model" refers to an artificial intelligence model that learns from large amounts of data and responds to or generates new data.

[0370] A "chatbot" is an automated response system for interacting with users.

[0371] A "communication interface" refers to a standardized connection method or protocol that allows different systems or devices to communicate.

[0372] "User interface" refers to the means and design by which a user interacts with a system.

[0373] "Personalized product recommendations" is a feature that suggests specific products based on a user's specific preferences and history.

[0374] This invention combines an emotion engine with a system that learns the digital information of deceased family members and recreates them as chatbots that can be interacted with 24 hours a day through a LINE account. The system is composed of a server, a device, and a user.

[0375] System Configuration

[0376] Server-side processing

[0377] 1. Data Collection

[0378] The server collects data such as electronic messages, electronic images, electronic videos, audio messages, social networking service logs, and letters from the deceased family members' electronic devices. This collection process involves using the API of the electronic devices to obtain the data and transferring it to the server. The collected data is then stored in a database and managed centrally.

[0379] 2. Data Preprocessing

[0380] The server preprocesses the collected data. Specifically, it normalizes text data (e.g., electronic messages, social media logs) and removes punctuation and special characters. It also performs optical character recognition (OCR) on electronic images and videos to extract text data. It also converts voice messages into text using speech recognition. This ensures that all data is converted into a unified text format.

[0381] 3. Model Training

[0382] The server trains a generative AI model based on the preprocessed text data. Specifically, the text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[0383] 4. Chatbot construction

[0384] The server uses a trained generative AI model to build a chatbot. The chatbot receives input from the user and generates an appropriate response. It also combines an emotion engine to recognize the user's emotions and provide a response that corresponds to the user's emotions.

[0385] 5. LINE account integration

[0386] The server integrates the chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. This integration allows users to converse with the chatbot through their LINE account.

[0387] 6. Product recommendation

[0388] The server provides personalized product recommendations through a user interface, identifying products related to the deceased through dialogue with the user and making appropriate recommendations.

[0389] User processing

[0390] When a user sends a message to the chatbot for a deceased family member through their LINE account, the message is sent to the server. The server's generative AI model analyzes the message and generates an appropriate response. The server's emotion engine also recognizes the user's emotions from the content of the message and generates a response based on those emotions. The generated response is then sent back to the user via the LINE API. At the same time, the system recommends products related to the deceased based on the content of the conversation.

[0391] Specific examples

[0392] For example, if a user asks the chatbot through their LINE account, "Today I want to make my mom's favorite dish. What would you like?", the following will happen:

[0393] 1. A user sends a message through their LINE account.

[0394] Message: "Today I want to make your mom's favorite dish. What would you like?"

[0395] 2. The device sends the message to the LINE server.

[0396] 3. The LINE server passes the message to the chatbot on the server.

[0397] 4. The server's generative AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the user's intention and generates a response based on information about the deceased's favorite dishes.

[0398] Sample response: "Mom loved spaghetti bolognese. Why don't you try making that?"

[0399] 5. The server generates a response and sends it back to the LINE server, which then sends it to the user's LINE account.

[0400] 6. The user receives a response from the server on their LINE account and can learn about their mother's favorite dishes. The chatbot will also continue to recommend related recipes and foods.

[0401] In this way, users can recreate the interactive experience of their deceased family members and enjoy a special shopping experience thanks to the emotion engine and chatbot functions.

[0402] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0403] Step 1:

[0404] The server collects data such as electronic messages, electronic images, electronic videos, audio messages, social network service logs, and letters from the electronic devices of deceased family members. As input, it uses the device's API to acquire this data and saves it in the server's database. As output, the collected data is stored in the database.

[0405] Step 2:

[0406] The server preprocesses the collected data. As input, it takes the raw data stored in the database. For text data, it normalizes it and removes punctuation and special characters. For electronic images and videos, it performs optical character recognition (OCR) to extract text data. For voice messages, it converts them to text using speech recognition. As output, it obtains data in a unified text format.

[0407] Step 3:

[0408] The server trains a generative AI model based on the preprocessed text data. The preprocessed text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses based on the data that reflect the deceased's language and thought patterns. The output is a trained generative AI model.

[0409] Step 4:

[0410] The server uses a trained generative AI model to build a chatbot. The trained generative AI model is used as input. The chatbot receives input from the user and generates an appropriate response. The output is a chatbot that can converse.

[0411] Step 5:

[0412] The server integrates the built chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. The input is the LINE Messaging API configuration information and the chatbot integration information. The output is that the user can converse with the chatbot through their LINE account.

[0413] Step 6:

[0414] The server makes product recommendations through dialogue with the user. Inputs include messages from the user and historical dialogue data. A generative AI model analyzes the user's messages and generates appropriate responses. An emotion engine recognizes the user's emotions and recommends products related to the deceased based on those emotions. The output is a personalized product recommendation message provided to the user via their LINE account.

[0415] Step 7:

[0416] The server receives feedback from users and records the feedback data to improve the system's performance. As input, it aggregates users' interaction ratings with the chatbot and their satisfaction with product recommendations. As output, the accuracy of the generative AI model and the recommendation system is improved based on the feedback.

[0417] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0418] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0419] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0420] [Second embodiment]

[0421] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0422] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0423] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0424] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0425] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0426] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0427] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0428] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0429] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0430] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0431] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0432] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0433] This invention relates to a system that learns the digital information of deceased family members and recreates it as a chatbot that can be interacted with 24 hours a day via a LINE account. This system is composed of elements that play the roles of a server, a terminal, and a user.

[0434] System Configuration

[0435] Server-side processing

[0436] 1. Data Collection

[0437] The server collects data from the deceased family member's electronic devices, including emails, photos, videos, voice messages, social networking logs, and letters. This data is organized into the deceased family member's digital legacy.

[0438] 2. Data Preprocessing

[0439] The collected data is preprocessed on the server. Specifically, text data is normalized, text is extracted from images and videos using optical character recognition (OCR), and voice messages are converted to text using speech recognition. This unifies the data from various formats into a unified text format.

[0440] 3. Model Training

[0441] The pre-processed data is sent to a server as a training dataset for a generative AI model, which is then trained to generate responses that mimic the language and thought patterns of the deceased family member.

[0442] 4. Chatbot construction

[0443] Using the trained generative AI model, the server builds a chatbot that responds appropriately to messages from users.

[0444] 5. LINE account integration

[0445] The chatbot is integrated into a LINE account through a communication interface. The server uses the LINE Messaging API to manage communication between the chatbot and the LINE platform.

[0446] User processing

[0447] When a user sends a message to the chatbot of a deceased family member through their LINE account, the message is sent to the server, where the generative AI model analyzes the message and generates an appropriate response, which is then sent back to the user via the LINE API.

[0448] Specific use cases

[0449] For example, consider the case where a user sends a message to a chatbot via their LINE account saying, "Show me some recent photos of my family."

[0450] 1. A user sends a message through their LINE account.

[0451] Message: "Show me a recent photo of your family."

[0452] 2. The device sends the message to the LINE server.

[0453] 3. The LINE server passes the message to the chatbot on the server.

[0454] 4. The server's generative AI model analyzes the user's message and generates an appropriate response.

[0455] Example response: "Here's a recent photo of my family." with a link or image file showing the photo

[0456] 5. The server sends the generated response through the LINE server.

[0457] 6. The user receives a response from the server in their LINE account and can view recent photos of their family.

[0458] This allows users to recreate conversations with their deceased family members, providing an experience that maintains an emotional connection. The system provides a way for bereaved families to receive emotional support while keeping the memory of their deceased family members a part of their lives.

[0459] The processing flow will be explained below.

[0460] Step 1:

[0461] The server collects data from the deceased family member's electronic devices, including emails, photos, videos, voice messages, social networking service logs, and letters. Specifically, it uses the API provided by the device to download the data and convert it into an appropriate data format. This allows various data to be aggregated in one place.

[0462] Step 2:

[0463] The server pre-processes the collected data, which includes the following steps:

[0464] Normalization of text data (emails, SNS logs), specifically removing punctuation, adjusting spaces, expanding slang and abbreviations, etc.

[0465] Perform optical character recognition (OCR) on image and video data to extract text.

[0466] Voice messages are converted into text using voice recognition technology.

[0467] Step 3:

[0468] The server trains a generative AI model based on the preprocessed text data. Specifically, the collected text data is tokenized and provided as input to the generative AI model. The model is tuned to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[0469] Step 4:

[0470] The server uses the trained generative AI model to build a chatbot. The built chatbot has the ability to receive input from the user and generate an appropriate response. The actual response that the chatbot generates depends on what the AI ​​model has learned.

[0471] Step 5:

[0472] The server integrates the chatbot into the communication interface. Specifically, it uses the LINE Messaging API to link the chatbot to a specific LINE account. This integration allows users to converse with the chatbot through their LINE account.

[0473] Step 6:

[0474] The user sends a message to the chatbot through their LINE account.

[0475] For example, if a user sends a message saying, "Show me the last photo of your family you took," the message is immediately transmitted to the LINE server.

[0476] Step 7:

[0477] The LINE server forwards the received message to the server, which then receives it and has the generative AI model analyze the message.

[0478] Step 8:

[0479] The server's generative AI model analyzes the received message and generates an appropriate response, such as "Here's the last photo of the family we took," along with a link to the photo.

[0480] Step 9:

[0481] The server sends the generated response back to the LINE server, which then sends it to the user's LINE account. The user can then view photos of the specified family member.

[0482] Through this series of steps, users can recreate the interaction experience with their deceased family members and maintain an emotional connection. The system provides a means for bereaved families to receive emotional support while maintaining a record of their deceased family members as part of their lives.

[0483] Example 1

[0484] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0485] In modern society, there is a lack of ways to continue to recreate memories and emotional connections with deceased family members. Existing technologies have difficulty integrating the diverse digital information of deceased family members to recreate a consistent interactive experience. Therefore, there is a need for a system that allows bereaved family members to receive emotional support while continuing to cherish memories of their deceased family members.

[0486] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0487] In this invention, the server includes means for collecting information such as emails, image data, video data, audio data, social network service logs, and document data from the information devices of the deceased family member, means for converting the collected data into text format and preprocessing it, means for training an artificial intelligence model generated based on the preprocessed data, means for constructing a dialogue agent using the trained generative artificial intelligence model, means for integrating the constructed dialogue agent into a communication interface, means for generating information based on the collected data and linking the dialogue agent to an external system, and means for analyzing the information and generating responses to inquiries from users. This makes it possible to recreate an emotional connection through natural dialogue using the digital information of the deceased family member, allowing the bereaved family to receive emotional support.

[0488] An "information device" is a device that has the function of generating, saving, and transferring email, image data, video data, audio data, social network service logs, document data, and the like.

[0489] A "data collection means" is a function or process for collecting specific digital data from information devices and transferring it to a server.

[0490] "Preprocessing means" refers to a function or process that analyzes collected digital data and converts the data into a text format for uniform formatting.

[0491] A "generated artificial intelligence model" is an AI technology that has the ability to learn from a specific dataset and generate responses in natural language.

[0492] A "dialogue agent" is a program or system that has the ability to respond to messages from a user using a generated artificial intelligence model.

[0493] A "communication interface" is a protocol or mechanism for sending and receiving information between an interactive agent and an external messaging platform.

[0494] "External system integration means" refers to a function that enables a conversational agent to operate in conjunction with other systems or platforms (e.g., messaging apps).

[0495] The "information analysis means" is a function or process for analyzing messages from users and generating optimal response content.

[0496] The "response generation means" is a function or process for generating an appropriate response to be provided to the user based on the analysis results.

[0497] This invention relates to a system that trains a generative AI model based on the digital information of deceased family members and recreates it as a chatbot that can interact with them 24 hours a day. Specific embodiments of this system are described below.

[0498] Data collection

[0499] The server collects various information from the information devices of deceased family members, including emails, image data, video data, audio data, social network service logs, and document data. Dedicated scripts and programs are used in the collection process, and this data is transferred to the server. Specifically, this occurs when a user uploads a photo folder on their smartphone using a dedicated application.

[0500] Data Preprocessing

[0501] The server analyzes the collected data and converts it into a text format to standardize it. It uses optical character recognition (OCR) technologies, such as Google Cloud Vision API and Tesseract, to normalize the text data and process image and video data. It also converts voice messages into text using speech recognition technologies, such as Google Speech-to-Text API. This ensures that all data is in a unified text format.

[0502] Model learning

[0503] The server uses the preprocessed data to train generative AI models, such as OpenAI's GPT-3 and ChatGPT models, which are trained to generate natural-sounding responses based on the deceased family member's vocabulary and thought patterns based on the text data.

[0504] Chatbot construction

[0505] The server uses the trained generative AI model to build a conversational agent, which is implemented as an API using the Python Flask framework, resulting in a chatbot that responds to messages from users.

[0506] LINE account integration

[0507] The server integrates the built conversational agent with a LINE account. Specifically, it uses the LINE Messaging API to manage communication between the chatbot and the LINE platform. When a user sends a message via LINE, the message is forwarded to the server, where the generative AI model analyzes it and generates an appropriate response. The generated response is then sent back to the user via the LINE API.

[0508] Examples of concrete examples and prompts

[0509] For example, here is a specific example of a user sending a request to a chatbot via their LINE account saying, "Show me recent photos of my family."

[0510] 1. A user sends a message through their LINE account.

[0511] Message: "Show me a recent photo of your family."

[0512] 2. The server's AI model analyzes this message and generates an appropriate response.

[0513] Example response: "Here's a recent photo of my family." with a link or image file showing the photo

[0514] Thus, a specific example of a prompt that a generative AI model can use to generate an appropriate response is "Hello, can you show me a recent family photo?" Because the model is trained on this prompt, it can provide a natural response to the user.

[0515] In this way, the bereaved can relive the conversations with their deceased loved ones and maintain emotional ties, which is a specific embodiment of the present invention.

[0516] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0517] Step 1:

[0518] Data collection

[0519] Input: Emails, image data, video data, audio data, social networking service logs, document data, etc. from the information devices of deceased family members.

[0520] Output: The collection of digital data transferred to the server.

[0521] Specific operation: The user uploads data from their smartphone or PC using a dedicated application. The data is collected by a dedicated script and transferred to the server. For example, the user selects a photo folder on their smartphone and presses the upload button, which sends the data to the server.

[0522] Step 2:

[0523] Data Preprocessing

[0524] Input: A collection of digital data transferred to the server.

[0525] Output: A normalized set of data converted to text format.

[0526] What it does: The server analyzes the collected data and converts it into text format. For example, it uses the Google Cloud Vision API to extract text from images and video data, and the Google Speech-to-Text API to convert audio data into text. Furthermore, it performs spell checking and grammar correction as part of the normalization process.

[0527] Step 3:

[0528] Model learning

[0529] Input: Preprocessed data in text format.

[0530] Output: A trained generative AI model.

[0531] How it works: The server inputs the preprocessed data into a generative AI model (e.g., GPT-3) and trains the model. Based on a large amount of text data, the model learns the language and thought patterns of the deceased family member. A dedicated training script is used for this training process. For example, preprocessed text data from emails or letters is input into the model, and training proceeds over a period of several hours to several days.

[0532] Step 4:

[0533] Chatbot construction

[0534] Input: A trained generative AI model.

[0535] Output: A chatbot API that can respond to user input.

[0536] How it works: The server uses a trained generative AI model to build a conversational agent. An API server is configured using Python's Flask framework, and the generative AI model is connected to this API. This creates a mechanism for the generative AI model to respond appropriately to external requests. For example, if an endpoint called " / message" is set up on the Flask server, when a message is POSTed there, the generative AI model will generate and return a response.

[0537] Step 5:

[0538] LINE account integration

[0539] Input: A LINE message from the user.

[0540] Output: The response of the generated AI model through a LINE account.

[0541] How it works: The server integrates the conversational agent into a LINE account using the LINE Messaging API. Create a channel on the LINE Developers site and set the API key in the Flask server. When a user sends a message from the LINE app, the message is forwarded to the server, where the generative AI model analyzes it and generates a response, which is then sent back to the user via the LINE API. For example, when a user sends "Hello" on LINE, the server passes the message to the generative AI model, which generates the response "Hello, how are you?" and sends it back to the LINE app.

[0542] Step 6:

[0543] User interaction

[0544] Input: A message from the user.

[0545] Output: The response message generated by the generative AI model.

[0546] Specific operation: A user sends a message to their LINE account. The device sends the message to the LINE server, which passes it on to the server's chatbot. The generative AI model analyzes the message, generates an appropriate response, and sends it back to the user via the LINE API. The user receives the response in their LINE account and can continue the conversation. For example, if a user sends "Show me some recent photos of your family," the server retrieves the latest photo from the database and sends it along with the response "Here are some recent photos of your family."

[0547] (Application example 1)

[0548] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0549] In recent years, bereaved families have increasingly sought emotional support and psychological healing, but existing technologies have limited the means to provide this digitally. Furthermore, with the spread of autonomous vehicles, how users spend their time in their cars has become an important issue. In response to this, there is a need to provide a safe means for them to communicate with their deceased family members.

[0550] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0551] In this invention, the server includes: means for collecting data such as emails, images, videos, voice messages, social network logs, and letters from the electronic devices of the deceased family member; means for converting the collected data into text format and preprocessing it; means for training a generative AI model based on the preprocessed data; means for building a chatbot using the trained generative AI model; means for integrating the built chatbot into a communication interface; and means, mounted on the smart glasses, for recognizing voice input and sending user dialogue commands to the chatbot. This allows bereaved family members to experience an emotional connection with their deceased family member through dialogue with them in an autonomous vehicle and safely find healing.

[0552] "Electronic device" is a general term for devices that generate, store, transmit, or receive digital data.

[0553] "Data" refers to information such as emails, images, videos, voice messages, social network logs, letters, etc.

[0554] "Means of collection" refers to the process for extracting specific data from electronic devices and transferring it to a server, etc.

[0555] "Preprocessing means" refers to the process of converting collected data into a unified text format and organizing it in a way that makes it easier to analyze.

[0556] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses to user input based on data.

[0557] "Training" refers to the process of using preprocessed data to train a generative AI model to learn specific patterns.

[0558] A "chatbot" is a program that can understand natural language and respond to users through dialogue.

[0559] "Means of integration into a communication interface" means connecting the chatbot to a specific communication platform (e.g., LINE) and enabling it to send and receive messages.

[0560] "Smart glasses" are wearable devices that have built-in displays and sensors, display information to the user, and can be operated by voice input, touch input, etc.

[0561] "Means for recognizing voice input and transmitting user interaction commands to the chatbot" refers to the process of using a microphone and software built into the smart glasses to convert voice input into text and transmit that text to the chatbot.

[0562] An "autonomous vehicle" is a vehicle that uses sensors and AI to drive autonomously without the need for driver operation.

[0563] The system for implementing this invention aims to safely recreate conversations with deceased family members. Specifically, it collects data from the deceased family member's electronic devices, builds a chatbot using a generative AI model based on that data, and enables the user to interact with the chatbot through smart glasses.

[0564] Server-side processing

[0565] 1. Data Collection

[0566] The server collects the digital information of deceased family members (e.g., emails, images, videos, audio messages, social network logs, letters, etc.), allowing for a complete picture of their digital lives.

[0567] 2. Data Preprocessing

[0568] The collected data is preprocessed on the server. Specifically, text data is normalized, and text is extracted from images and videos using optical character recognition (OCR). Voice messages are then converted to text using speech recognition. This preprocessing process converts data from a variety of formats into a unified text format.

[0569] 3. Model Training

[0570] The preprocessed data is sent to a server as a training dataset for a generative AI model (e.g., GPT-3, BERT). The generative AI model learns from this dataset and is tuned to generate responses that mimic the language and thought patterns of the deceased family member.

[0571] 4. Chatbot construction

[0572] Using the trained generative AI model, the server builds a chatbot that can respond appropriately to messages from users.

[0573] 5. Communication interface integration

[0574] The constructed chatbot is integrated into a communication interface (e.g., LINE API), through which the server manages communication between the chatbot and the user.

[0575] User processing

[0576] 1. Voice input

[0577] Suppose a user is riding in a self-driving vehicle and wearing smart glasses. When the user speaks, "Mom, what's the weather like today?", the smart glasses recognize this as voice input. The smart glasses have a built-in microphone that captures this voice input.

[0578] 2. Voice Recognition

[0579] Software installed on the smart glasses (e.g., voice recognition software using Google Cloud API) converts this voice input into text, allowing the user's commands to be captured in text format.

[0580] 3. Send a message

[0581] The captured text is sent from the smart glasses via a communication interface to a server, where a generative AI model analyzes the user's message and generates an appropriate response.

[0582] 4. Receiving a Response

[0583] The generated response is then sent back to the smart glasses via the communication interface and presented to the user in voice or text format, for example, "The weather is sunny today, it looks like it's going to be a nice day."

[0584] Specific examples

[0585] As a concrete example, consider the case where a user in a self-driving vehicle asks, "Mom, how's the weather today?" The smart glasses recognize the speech, convert it into text, and then send it to the server. The server's generative AI model analyzes the message and generates a response to the user: "The weather is sunny today. It looks like it's going to be a nice day."

[0586] Prompt Sentence Examples

[0587] Speech prompt: "Mom, how's the weather today?"

[0588] Chatbot prompt: "A 24-hour conversational chatbot that responds based on digital information about deceased family members."

[0589] This allows users to safely interact with deceased family members from within their self-driving vehicles and receive emotional support.

[0590] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0591] Step 1:

[0592] The server collects data from the deceased family member's electronic devices, including emails, images, videos, audio messages, social network logs, and letters. The data is transferred using APIs and file transfer protocols. The input is various digital data, and the output is a unified data set stored on the server.

[0593] Step 2:

[0594] The server preprocesses the collected data by normalizing the text data, extracting text from images and videos using optical character recognition (OCR), and converting voice messages into text using speech recognition software. The input is the collected data, and the output is normalized text data.

[0595] Step 3:

[0596] The server trains a generative AI model based on the preprocessed data. Using the preprocessed data as a training dataset, an AI algorithm (e.g., GPT-3, BERT) learns language and thought patterns. The input is normalized text data, and the output is a trained generative AI model.

[0597] Step 4:

[0598] The server builds a chatbot using a trained generative AI model, which enables the AI ​​model to generate natural and appropriate responses to user input. The input is the trained generative AI model, and the output is the chatbot.

[0599] Step 5:

[0600] The server integrates the constructed chatbot into a communication interface (e.g., LINE API), allowing users to interact with the chatbot through their LINE accounts. The input is connection information between the chatbot and the communication interface, and the output is the established communication state.

[0601] Step 6:

[0602] A user gets into an autonomous vehicle and puts on smart glasses. The user speaks to the vehicle, saying, "Mom, what's the weather like today?" The input is the user's voice command, and the output is the audio data of the voice command.

[0603] Step 7:

[0604] The smart glasses use a built-in microphone to recognize the user's voice and capture acoustic data, then use speech recognition software to convert the speech into text, with the input being acoustic data and the output being text data.

[0605] Step 8:

[0606] The smart glasses send the acquired text data to the server through the communication interface. The input is the text data and the output is the transmitted text message.

[0607] Step 9:

[0608] The server uses a generative AI model to analyze the received text message and generate an appropriate response. The input is the text message from the user, and the output is the generated response text.

[0609] Step 10:

[0610] The server sends the generated response to the smart glasses through the communication interface, where the input is the generated response text and the output is the text or audio displayed on the smart glasses.

[0611] Step 11:

[0612] The smart glasses provide the received response to the user in voice or text format. The input is the transmitted response text, and the output is the information presented to the user. Specifically, the response may be something like, "The weather is sunny today, it looks like it's going to be a nice day."

[0613] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0614] This invention combines an emotion engine with a system that learns the digital information of deceased family members and recreates them as chatbots that can be interacted with 24 hours a day via LINE accounts. The system is composed of elements that play the roles of a server, a device, and a user.

[0615] System Configuration

[0616] Server-side processing

[0617] 1. Data Collection

[0618] The server collects data such as emails, photos, videos, voice messages, social networking service logs, and letters from the deceased family members' electronic devices. This involves using the device's API to obtain the data and transferring it to the server. The collected data is then stored in a database and managed centrally.

[0619] 2. Data Preprocessing

[0620] The server preprocesses the collected data. Specifically, it normalizes text data (e.g., emails, social media logs) and removes punctuation and special characters. It also performs optical character recognition (OCR) on images and videos to extract text data. Voice messages are converted to text using speech recognition. This ensures that all data is converted into a unified text format.

[0621] 3. Model Training

[0622] The server trains a generative AI model based on the preprocessed text data. Specifically, the text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[0623] 4. Chatbot construction

[0624] The server uses a trained generative AI model to build a chatbot. The chatbot receives input from the user and generates an appropriate response. The server also combines an emotion engine to recognize the user's emotions and provide a response according to the user's emotions.

[0625] 5. LINE account integration

[0626] The server integrates the chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. This integration allows users to converse with the chatbot through their LINE account.

[0627] User processing

[0628] When a user sends a message to the chatbot for a deceased family member through their LINE account, the message is sent to the server. The server's generative AI model analyzes the message and generates an appropriate response. The server's emotion engine recognizes the user's emotion from the message content and generates a response based on that emotion. The generated response is then sent back to the user via the LINE API.

[0629] Specific use cases

[0630] For example, if a user sends "I'm tired today" to the chatbot through their LINE account, the following will happen:

[0631] 1. A user sends a message through their LINE account.

[0632] Message: "I'm tired today."

[0633] 2. The device sends the message to the LINE server.

[0634] 3. The LINE server passes the message to the chatbot on the server.

[0635] 4. The server's AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the emotion "tired" and generates a response that includes words of comfort for fatigue.

[0636] Example response: "Good work. Have a good rest today."

[0637] 5. The server generates a response and sends it back to the LINE server, which then sends it to the user's LINE account.

[0638] 6. The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[0639] In this way, users can recreate the interaction experience with their deceased family members and, with the help of the emotion engine, receive more advanced emotional support, providing a means for bereaved families to receive better emotional support and cherish the memories of their deceased family members.

[0640] The processing flow will be explained below.

[0641] Step 1:

[0642] The server collects data such as emails, photos, videos, voice messages, social networking service logs, and letters from the electronic devices of deceased family members. Specifically, it retrieves and aggregates various digital data stored on the electronic devices using APIs and file transfer protocols.

[0643] Step 2:

[0644] The server pre-processes the collected data, which includes:

[0645] Normalize text data (e.g., emails and social media logs) to remove punctuation and special characters.

[0646] Optical character recognition (OCR) is used on image and video data to extract text information.

[0647] Voice messages are converted into text using voice recognition technology, which ensures that all data is converted into a unified text format.

[0648] Step 3:

[0649] The server trains a generative AI model based on the preprocessed text data, using a large dataset to train the model to reflect the deceased family member's vocabulary and thought patterns, enabling the AI ​​model to generate appropriate natural language responses.

[0650] Step 4:

[0651] The server uses the trained generative AI model to build a chatbot, which receives messages from users and generates appropriate responses.

[0652] Step 5:

[0653] The server recognizes the user's emotions using an emotion engine. Specifically, the emotion engine analyzes the input message and identifies the user's emotions (e.g., joy, sadness, anger, fatigue, etc.) using an emotion analysis algorithm.

[0654] Step 6:

[0655] Based on the analysis results of the emotion engine, the server uses a generative AI model to generate a response based on the emotion. This response is tailored to the user's current emotional state. For example, if a user sends a message saying, "I'm tired today," the emotion engine will recognize the emotion "fatigue," and the generative AI model will generate a comforting message such as, "Thank you for your hard work. Take a good rest today."

[0656] Step 7:

[0657] The server integrates the built chatbot into the communication interface. Specifically, it uses the LINE Messaging API to link the chatbot to a LINE account. This integration allows users to converse with the chatbot through their LINE account.

[0658] Step 8:

[0659] The user sends a message to the chatbot of their deceased family member through their LINE account. At this time, the user's message is sent to the server via the LINE server.

[0660] Step 9:

[0661] The server analyzes the message received from the LINE server using the generative AI model and generates an appropriate response. This includes the process where the emotion engine recognizes the user's emotions and generates a response that corresponds to those emotions.

[0662] Step 10:

[0663] The server sends the generated response back to the LINE server, which then sends it to the user's LINE account.

[0664] Step 11:

[0665] The user receives a response from the server on their LINE account and checks its contents. For example, they may receive a message such as "Thank you for your hard work. Please take a good rest today," providing emotional support.

[0666] This series of steps allows users to re-create conversations with deceased family members and, with the help of an emotion engine, receive more personalized emotional support.

[0667] Example 2

[0668] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0669] With limited means to store and relive memories of deceased family members, bereaved families find it difficult to receive emotional support or comfort through interactive experiences. To solve this problem, an advanced dialogue system that utilizes the digital information of deceased family members and also provides emotional support is needed.

[0670] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0671] In this invention, the server includes means for collecting data such as emails, images, videos, voice communications, social network service records, and letters from the information devices of the deceased family member, means for converting the collected data into text format and performing preprocessing, means for training a generative AI model based on the preprocessed data, means for constructing a dialogue system using the trained generative AI model, means for integrating the constructed dialogue system into a communication interface, emotion engine means for analyzing messages from users and recognizing their emotions, and means for generating responses based on the recognition results of the emotion engine. This allows the bereaved family to relive conversations with their deceased family member and receive support tailored to their emotions.

[0672] "Information equipment" is a general term for devices that can generate, store, manage, and transmit data such as emails, images, videos, voice communications, social network service records, and letters.

[0673] "Data collection means" refers to the functions and processes that automatically or manually acquire data from various information devices and store it on a central server.

[0674] "Data preprocessing measures" are processes such as normalization, removal of punctuation and special characters, optical character recognition (OCR) and speech recognition to convert collected data into a unified text format.

[0675] A "generative AI model" is an algorithm or artificial intelligence (AI) system that is trained based on collected text data to understand specific patterns and contexts and generate appropriate responses.

[0676] A "dialogue system" is a software application built using generative AI models to engage in dialogue with users and provide appropriate responses to their questions and comments.

[0677] A "communication interface" is a mechanism for connecting a dialogue system with a user, and often uses messaging platforms such as LINE or email.

[0678] "Emotion engine means" refers to algorithms and technologies for analyzing messages from users and recognizing the emotions contained in the content.

[0679] "Response generation means" refers to a process or function for generating an appropriate response message based on the emotion recognized by the emotion engine.

[0680] "Preprocessed data" refers to raw data collected that has been converted into text format and processed to a state suitable for normalization and analysis.

[0681] This invention relates to a system for building chatbots that can converse in real time based on the digital information of deceased family members. This system includes elements of a server, a terminal, and a user, and by combining an emotion engine, provides more advanced emotional support.

[0682] First, the server collects data from the deceased family member's electronic devices. Specifically, the hardware used utilizes data collection APIs (e.g., Gmail API, Google Photos API) to obtain data such as emails, photos, videos, voice messages, social networking service records, and letters. This data is then stored in a database on the server and managed centrally.

[0683] Next, the server preprocesses the collected data. The collected data is not suitable for analysis or training generative AI models as it is, so it must be converted into text format. Text data is normalized and punctuation and special characters are removed, and image and video data undergoes optical character recognition (OCR). Audio data is then converted into text using speech recognition technology (e.g., Google Speech-to-Text API). This preprocessing ensures that all data is converted into a unified text format.

[0684] The server then trains a generative AI model using the preprocessed text data. The generative AI model uses a natural language processing library (e.g., Transformers). This process involves providing the preprocessed data as input to the AI ​​model, training it to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[0685] Once the learning is complete, the server uses the trained generative AI model to build a chatbot. The chatbot receives messages from users and generates appropriate responses. Furthermore, by incorporating an emotion engine (e.g., Sentiment Analysis) to recognize the user's emotions, the chatbot provides responses that correspond to the user's emotions.

[0686] The server then integrates the chatbot into the communication interface. This process involves linking the chatbot to a LINE account using the LINE Messaging API. This integration allows users to interact with the chatbot through the LINE platform.

[0687] When a user sends a message through their LINE account, the message is sent via their device to the LINE server, where it is then forwarded to a chatbot on the server. The generative AI model on the server analyzes the user's message and generates an appropriate response. The emotion engine recognizes the user's emotion from the content of the message and generates a response based on that emotion. The generated response is then sent back to the user via the LINE API.

[0688] Specific examples

[0689] For example, if a user sends "I'm tired today" to a chatbot through their LINE account, the process will be as follows:

[0690] 1. A user sends a message saying, "I'm tired today."

[0691] 2. The device sends the message to the LINE server.

[0692] 3. The LINE server forwards the message to the chatbot on the server.

[0693] 4. The server's AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the emotion "tired" and generates a response that includes words of comfort for fatigue.

[0694] Example response: "Good work. Have a good rest today."

[0695] 5. The server generates a response and sends it back to the LINE server.

[0696] 6. The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[0697] In this way, users can recreate the interaction experience with their deceased family members and, with the help of the emotion engine, receive higher levels of emotional support.

[0698] Prompt Sentence Examples

[0699] Here are some examples of input prompts for generative AI models:

[0700] "I've been so busy and tired today. Could you please give me some words of encouragement?"

[0701] "I've been feeling a bit down lately...what should I do?"

[0702] "I'm really looking forward to talking to you."

[0703] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0704] Step 1:

[0705] Data collection

[0706] The server collects data such as emails, images, videos, voice communications, social networking service records, and letters from the information devices of deceased family members.

[0707] Specifically, it uses the Gmail API to collect emails, the Google Photos API to download photos and videos, and retrieves voice messages from the device's storage.

[0708] Input: Data retrieval request from information device.

[0709] Output: Raw data (emails, images, videos, audio messages, etc.) stored in a server database.

[0710] Step 2:

[0711] Data Preprocessing

[0712] The server preprocesses the collected raw data.

[0713] Text data is normalized to remove punctuation and special characters, image and video data is converted to text using OCR (Optical Character Recognition), and voice messages are converted to text using speech recognition technology.

[0714] As a concrete example, we use the Google Speech-to-Text API to convert voice messages into text and Tesseract OCR to extract text from images.

[0715] Input: Raw data stored in the server database.

[0716] Output: Preprocessed text data.

[0717] Step 3:

[0718] Model learning

[0719] The server uses the preprocessed text data to train a generative AI model.

[0720] Use a natural language processing library (e.g., Transformers) to train your AI model using preprocessed data.

[0721] Specifically, the collected data sets are batch processed and input into the AI ​​model sequentially.

[0722] Input: Preprocessed text data.

[0723] Output: A trained generative AI model.

[0724] Step 4:

[0725] Chatbot construction

[0726] The server builds a chatbot using a trained generative AI model.

[0727] Chatbots have the ability to analyze messages from users and generate appropriate responses. Furthermore, by incorporating an emotion engine (e.g., Sentiment Analysis), they can recognize the user's emotions and provide responses that correspond to those emotions.

[0728] Specifically, it incorporates a sentiment analysis library to add emotional elements to the generated responses.

[0729] Input: Trained generative AI model and emotion engine.

[0730] Output: Available chatbot systems.

[0731] Step 5:

[0732] LINE account integration

[0733] The server integrates the built chatbot into the communication interface.

[0734] Use the LINE Messaging API to link your chatbot to your LINE account, allowing users to interact with your chatbot through the LINE Platform.

[0735] Specifically, the API endpoint is set using the channel access token obtained from the LINE Developer Console and registered in the chatbot's message receiving mechanism.

[0736] Input: Chatbot system and LINE Messaging API.

[0737] Output: Chatbot linked to LINE account.

[0738] Step 6:

[0739] Sending a message and generating a response

[0740] When a user sends a message through their LINE account, the message is sent via the device to the LINE server, and then forwarded to a chatbot on the server.

[0741] The server's generative AI model analyzes the user's message and generates an appropriate response. The emotion engine recognizes the user's emotion from the message content and generates a response based on that emotion.

[0742] Example: When a user sends "I'm tired today," the emotion engine recognizes the emotion "tired" and generates a response like "Great work. Take a good rest today."

[0743] Input: A message from the user.

[0744] Output: An appropriate response message that takes emotion into consideration.

[0745] Step 7:

[0746] Returning a response

[0747] The server generates a response message and sends it back to the LINE server, which then sends it to the user's LINE account.

[0748] The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[0749] Specifically, the generated response message is sent as an API request to the LINE Messaging API and delivered to the user's LINE account.

[0750] Input: The server-generated response message.

[0751] Output: A response message that will be displayed on the user's LINE account.

[0752] (Application example 2)

[0753] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0754] In today's world, the means for remembering the deceased are limited, making it difficult to maintain emotional connections. Furthermore, the means for identifying and purchasing products and mementos related to the deceased are not highly personalized. As a result, it is difficult for bereaved families to experience a memorial experience that is emotionally supported. This invention aims to solve this problem by providing a system that learns the digital information of the deceased, provides emotional support through dialogue with the user, and recommends products related to the deceased.

[0755] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0756] In this invention, the server includes: means for collecting data such as electronic messages, electronic images, electronic videos, audio messages, social network service logs, and letters from the electronic devices of the deceased family member; means for converting the collected data into text format and preprocessing it; means for training a generative AI model based on the preprocessed data; means for building a chatbot using the trained generative AI model; means for integrating the built chatbot into a communication interface; and means for providing personalized product recommendations through the user interface. This allows the user to maintain an emotional connection with the deceased through conversations with the deceased, receive recommendations for products related to the deceased, and enjoy a special shopping experience.

[0757] An "electronic message" is text information sent or received in digital form, such as an email or social media message.

[0758] "Electronic image" refers to a still image or photograph stored in digital format.

[0759] "Electronic moving images" refers to moving images or video clips stored in digital format.

[0760] A "voice message" is a message that includes recorded voice data.

[0761] "Social Network Service Logs" refers to records of user activities and interactions on social networking services.

[0762] A "letter" refers to a written communication, either on paper or digitally.

[0763] A "text format" is a format that describes information using letters and symbols.

[0764] "Preprocessing" refers to the process of converting and organizing data into a form suitable for further processing and analysis.

[0765] A "generative AI model" refers to an artificial intelligence model that learns from large amounts of data and responds to or generates new data.

[0766] A "chatbot" is an automated response system for interacting with users.

[0767] A "communication interface" refers to a standardized connection method or protocol that allows different systems or devices to communicate.

[0768] "User interface" refers to the means and design by which a user interacts with a system.

[0769] "Personalized product recommendations" is a feature that suggests specific products based on a user's specific preferences and history.

[0770] This invention combines an emotion engine with a system that learns the digital information of deceased family members and recreates them as chatbots that can be interacted with 24 hours a day through a LINE account. The system is composed of a server, a device, and a user.

[0771] System Configuration

[0772] Server-side processing

[0773] 1. Data Collection

[0774] The server collects data such as electronic messages, electronic images, electronic videos, audio messages, social networking service logs, and letters from the deceased family members' electronic devices. This collection process involves using the API of the electronic devices to obtain the data and transferring it to the server. The collected data is then stored in a database and managed centrally.

[0775] 2. Data Preprocessing

[0776] The server preprocesses the collected data. Specifically, it normalizes text data (e.g., electronic messages, social media logs) and removes punctuation and special characters. It also performs optical character recognition (OCR) on electronic images and videos to extract text data. It also converts voice messages into text using speech recognition. This ensures that all data is converted into a unified text format.

[0777] 3. Model Training

[0778] The server trains a generative AI model based on the preprocessed text data. Specifically, the text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[0779] 4. Chatbot construction

[0780] The server uses a trained generative AI model to build a chatbot. The chatbot receives input from the user and generates an appropriate response. It also combines an emotion engine to recognize the user's emotions and provide a response that corresponds to the user's emotions.

[0781] 5. LINE account integration

[0782] The server integrates the chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. This integration allows users to converse with the chatbot through their LINE account.

[0783] 6. Product recommendation

[0784] The server provides personalized product recommendations through a user interface, identifying products related to the deceased through dialogue with the user and making appropriate recommendations.

[0785] User processing

[0786] When a user sends a message to the chatbot for a deceased family member through their LINE account, the message is sent to the server. The server's generative AI model analyzes the message and generates an appropriate response. The server's emotion engine also recognizes the user's emotions from the content of the message and generates a response based on those emotions. The generated response is then sent back to the user via the LINE API. At the same time, the system recommends products related to the deceased based on the content of the conversation.

[0787] Specific examples

[0788] For example, if a user asks the chatbot through their LINE account, "Today I want to make my mom's favorite dish. What would you like?", the following will happen:

[0789] 1. A user sends a message through their LINE account.

[0790] Message: "Today I want to make your mom's favorite dish. What would you like?"

[0791] 2. The device sends the message to the LINE server.

[0792] 3. The LINE server passes the message to the chatbot on the server.

[0793] 4. The server's generative AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the user's intention and generates a response based on information about the deceased's favorite dishes.

[0794] Sample response: "Mom loved spaghetti bolognese. Why don't you try making that?"

[0795] 5. The server generates a response and sends it back to the LINE server, which then sends it to the user's LINE account.

[0796] 6. The user receives a response from the server on their LINE account and can learn about their mother's favorite dishes. The chatbot will also continue to recommend related recipes and foods.

[0797] In this way, users can recreate the interactive experience of their deceased family members and enjoy a special shopping experience thanks to the emotion engine and chatbot functions.

[0798] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0799] Step 1:

[0800] The server collects data such as electronic messages, electronic images, electronic videos, audio messages, social network service logs, and letters from the electronic devices of deceased family members. As input, it uses the device's API to acquire this data and saves it in the server's database. As output, the collected data is stored in the database.

[0801] Step 2:

[0802] The server preprocesses the collected data. As input, it takes the raw data stored in the database. For text data, it normalizes it and removes punctuation and special characters. For electronic images and videos, it performs optical character recognition (OCR) to extract text data. For voice messages, it converts them to text using speech recognition. As output, it obtains data in a unified text format.

[0803] Step 3:

[0804] The server trains a generative AI model based on the preprocessed text data. The preprocessed text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses based on the data that reflect the deceased's language and thought patterns. The output is a trained generative AI model.

[0805] Step 4:

[0806] The server uses a trained generative AI model to build a chatbot. The trained generative AI model is used as input. The chatbot receives input from the user and generates an appropriate response. The output is a chatbot that can converse.

[0807] Step 5:

[0808] The server integrates the built chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. The input is the LINE Messaging API configuration information and the chatbot integration information. The output is that the user can converse with the chatbot through their LINE account.

[0809] Step 6:

[0810] The server makes product recommendations through dialogue with the user. Inputs include messages from the user and historical dialogue data. A generative AI model analyzes the user's messages and generates appropriate responses. An emotion engine recognizes the user's emotions and recommends products related to the deceased based on those emotions. The output is a personalized product recommendation message provided to the user via their LINE account.

[0811] Step 7:

[0812] The server receives feedback from users and records the feedback data to improve the system's performance. As input, it aggregates users' interaction ratings with the chatbot and their satisfaction with product recommendations. As output, the accuracy of the generative AI model and the recommendation system is improved based on the feedback.

[0813] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0814] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0815] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0816] [Third embodiment]

[0817] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0818] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0819] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0820] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0821] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0822] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0823] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0824] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0825] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0826] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0827] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0828] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0829] This invention relates to a system that learns the digital information of deceased family members and recreates it as a chatbot that can be interacted with 24 hours a day via a LINE account. This system is composed of elements that play the roles of a server, a terminal, and a user.

[0830] System Configuration

[0831] Server-side processing

[0832] 1. Data Collection

[0833] The server collects data from the deceased family member's electronic devices, including emails, photos, videos, voice messages, social networking logs, and letters. This data is organized into the deceased family member's digital legacy.

[0834] 2. Data Preprocessing

[0835] The collected data is preprocessed on the server. Specifically, text data is normalized, text is extracted from images and videos using optical character recognition (OCR), and voice messages are converted to text using speech recognition. This unifies the data from various formats into a unified text format.

[0836] 3. Model Training

[0837] The pre-processed data is sent to a server as a training dataset for a generative AI model, which is then trained to generate responses that mimic the language and thought patterns of the deceased family member.

[0838] 4. Chatbot construction

[0839] Using the trained generative AI model, the server builds a chatbot that responds appropriately to messages from users.

[0840] 5. LINE account integration

[0841] The chatbot is integrated into a LINE account through a communication interface. The server uses the LINE Messaging API to manage communication between the chatbot and the LINE platform.

[0842] User processing

[0843] When a user sends a message to the chatbot of a deceased family member through their LINE account, the message is sent to the server, where the generative AI model analyzes the message and generates an appropriate response, which is then sent back to the user via the LINE API.

[0844] Specific use cases

[0845] For example, consider the case where a user sends a message to a chatbot via their LINE account saying, "Show me some recent photos of my family."

[0846] 1. A user sends a message through their LINE account.

[0847] Message: "Show me a recent photo of your family."

[0848] 2. The device sends the message to the LINE server.

[0849] 3. The LINE server passes the message to the chatbot on the server.

[0850] 4. The server's generative AI model analyzes the user's message and generates an appropriate response.

[0851] Example response: "Here's a recent photo of my family." with a link or image file showing the photo

[0852] 5. The server sends the generated response through the LINE server.

[0853] 6. The user receives a response from the server in their LINE account and can view recent photos of their family.

[0854] This allows users to recreate conversations with their deceased family members, providing an experience that maintains an emotional connection. The system provides a way for bereaved families to receive emotional support while keeping the memory of their deceased family members a part of their lives.

[0855] The processing flow will be explained below.

[0856] Step 1:

[0857] The server collects data from the deceased family member's electronic devices, including emails, photos, videos, voice messages, social networking service logs, and letters. Specifically, it uses the API provided by the device to download the data and convert it into an appropriate data format. This allows various data to be aggregated in one place.

[0858] Step 2:

[0859] The server pre-processes the collected data, which includes the following steps:

[0860] Normalization of text data (emails, SNS logs), specifically removing punctuation, adjusting spaces, expanding slang and abbreviations, etc.

[0861] Perform optical character recognition (OCR) on image and video data to extract text.

[0862] Voice messages are converted into text using voice recognition technology.

[0863] Step 3:

[0864] The server trains a generative AI model based on the preprocessed text data. Specifically, the collected text data is tokenized and provided as input to the generative AI model. The model is tuned to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[0865] Step 4:

[0866] The server uses the trained generative AI model to build a chatbot. The built chatbot has the ability to receive input from the user and generate an appropriate response. The actual response that the chatbot generates depends on what the AI ​​model has learned.

[0867] Step 5:

[0868] The server integrates the chatbot into the communication interface. Specifically, it uses the LINE Messaging API to link the chatbot to a specific LINE account. This integration allows users to converse with the chatbot through their LINE account.

[0869] Step 6:

[0870] The user sends a message to the chatbot through their LINE account.

[0871] For example, if a user sends a message saying, "Show me the last photo of your family you took," the message is immediately transmitted to the LINE server.

[0872] Step 7:

[0873] The LINE server forwards the received message to the server, which then receives it and has the generative AI model analyze the message.

[0874] Step 8:

[0875] The server's generative AI model analyzes the received message and generates an appropriate response, such as "Here's the last photo of the family we took," along with a link to the photo.

[0876] Step 9:

[0877] The server sends the generated response back to the LINE server, which then sends it to the user's LINE account. The user can then view photos of the specified family member.

[0878] Through this series of steps, users can recreate the interaction experience with their deceased family members and maintain an emotional connection. The system provides a means for bereaved families to receive emotional support while maintaining a record of their deceased family members as part of their lives.

[0879] Example 1

[0880] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0881] In modern society, there is a lack of ways to continue to recreate memories and emotional connections with deceased family members. Existing technologies have difficulty integrating the diverse digital information of deceased family members to recreate a consistent interactive experience. Therefore, there is a need for a system that allows bereaved family members to receive emotional support while continuing to cherish memories of their deceased family members.

[0882] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0883] In this invention, the server includes means for collecting information such as emails, image data, video data, audio data, social network service logs, and document data from the information devices of the deceased family member, means for converting the collected data into text format and preprocessing it, means for training an artificial intelligence model generated based on the preprocessed data, means for constructing a dialogue agent using the trained generative artificial intelligence model, means for integrating the constructed dialogue agent into a communication interface, means for generating information based on the collected data and linking the dialogue agent to an external system, and means for analyzing the information and generating responses to inquiries from users. This makes it possible to recreate an emotional connection through natural dialogue using the digital information of the deceased family member, allowing the bereaved family to receive emotional support.

[0884] An "information device" is a device that has the function of generating, saving, and transferring email, image data, video data, audio data, social network service logs, document data, and the like.

[0885] A "data collection means" is a function or process for collecting specific digital data from information devices and transferring it to a server.

[0886] "Preprocessing means" refers to a function or process that analyzes collected digital data and converts the data into a text format for uniform formatting.

[0887] A "generated artificial intelligence model" is an AI technology that has the ability to learn from a specific dataset and generate responses in natural language.

[0888] A "dialogue agent" is a program or system that has the ability to respond to messages from a user using a generated artificial intelligence model.

[0889] A "communication interface" is a protocol or mechanism for sending and receiving information between an interactive agent and an external messaging platform.

[0890] "External system integration means" refers to a function that enables a conversational agent to operate in conjunction with other systems or platforms (e.g., messaging apps).

[0891] The "information analysis means" is a function or process for analyzing messages from users and generating optimal response content.

[0892] The "response generation means" is a function or process for generating an appropriate response to be provided to the user based on the analysis results.

[0893] This invention relates to a system that trains a generative AI model based on the digital information of deceased family members and recreates it as a chatbot that can interact with them 24 hours a day. Specific embodiments of this system are described below.

[0894] Data collection

[0895] The server collects various information from the information devices of deceased family members, including emails, image data, video data, audio data, social network service logs, and document data. Dedicated scripts and programs are used in the collection process, and this data is transferred to the server. Specifically, this occurs when a user uploads a photo folder on their smartphone using a dedicated application.

[0896] Data Preprocessing

[0897] The server analyzes the collected data and converts it into a text format to standardize it. It uses optical character recognition (OCR) technologies, such as Google Cloud Vision API and Tesseract, to normalize the text data and process image and video data. It also converts voice messages into text using speech recognition technologies, such as Google Speech-to-Text API. This ensures that all data is in a unified text format.

[0898] Model learning

[0899] The server uses the preprocessed data to train generative AI models, such as OpenAI's GPT-3 and ChatGPT models, which are trained to generate natural-sounding responses based on the deceased family member's vocabulary and thought patterns based on the text data.

[0900] Chatbot construction

[0901] The server uses the trained generative AI model to build a conversational agent, which is implemented as an API using the Python Flask framework, resulting in a chatbot that responds to messages from users.

[0902] LINE account integration

[0903] The server integrates the built conversational agent with a LINE account. Specifically, it uses the LINE Messaging API to manage communication between the chatbot and the LINE platform. When a user sends a message via LINE, the message is forwarded to the server, where the generative AI model analyzes it and generates an appropriate response. The generated response is then sent back to the user via the LINE API.

[0904] Examples of concrete examples and prompts

[0905] For example, here is a specific example of a user sending a request to a chatbot via their LINE account saying, "Show me recent photos of my family."

[0906] 1. A user sends a message through their LINE account.

[0907] Message: "Show me a recent photo of your family."

[0908] 2. The server's AI model analyzes this message and generates an appropriate response.

[0909] Example response: "Here's a recent photo of my family." with a link or image file showing the photo

[0910] Thus, a specific example of a prompt that a generative AI model can use to generate an appropriate response is "Hello, can you show me a recent family photo?" Because the model is trained on this prompt, it can provide a natural response to the user.

[0911] In this way, the bereaved can relive the conversations with their deceased loved ones and maintain emotional ties, which is a specific embodiment of the present invention.

[0912] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0913] Step 1:

[0914] Data collection

[0915] Input: Emails, image data, video data, audio data, social networking service logs, document data, etc. from the information devices of deceased family members.

[0916] Output: The collection of digital data transferred to the server.

[0917] Specific operation: The user uploads data from their smartphone or PC using a dedicated application. The data is collected by a dedicated script and transferred to the server. For example, the user selects a photo folder on their smartphone and presses the upload button, which sends the data to the server.

[0918] Step 2:

[0919] Data Preprocessing

[0920] Input: A collection of digital data transferred to the server.

[0921] Output: A normalized set of data converted to text format.

[0922] What it does: The server analyzes the collected data and converts it into text format. For example, it uses the Google Cloud Vision API to extract text from images and video data, and the Google Speech-to-Text API to convert audio data into text. Furthermore, it performs spell checking and grammar correction as part of the normalization process.

[0923] Step 3:

[0924] Model learning

[0925] Input: Preprocessed data in text format.

[0926] Output: A trained generative AI model.

[0927] How it works: The server inputs the preprocessed data into a generative AI model (e.g., GPT-3) and trains the model. Based on a large amount of text data, the model learns the language and thought patterns of the deceased family member. A dedicated training script is used for this training process. For example, preprocessed text data from emails or letters is input into the model, and training proceeds over a period of several hours to several days.

[0928] Step 4:

[0929] Chatbot construction

[0930] Input: A trained generative AI model.

[0931] Output: A chatbot API that can respond to user input.

[0932] How it works: The server uses a trained generative AI model to build a conversational agent. An API server is configured using Python's Flask framework, and the generative AI model is connected to this API. This creates a mechanism for the generative AI model to respond appropriately to external requests. For example, if an endpoint called " / message" is set up on the Flask server, when a message is POSTed there, the generative AI model will generate and return a response.

[0933] Step 5:

[0934] LINE account integration

[0935] Input: A LINE message from the user.

[0936] Output: The response of the generated AI model through a LINE account.

[0937] How it works: The server integrates the conversational agent into a LINE account using the LINE Messaging API. Create a channel on the LINE Developers site and set the API key in the Flask server. When a user sends a message from the LINE app, the message is forwarded to the server, where the generative AI model analyzes it and generates a response, which is then sent back to the user via the LINE API. For example, when a user sends "Hello" on LINE, the server passes the message to the generative AI model, which generates the response "Hello, how are you?" and sends it back to the LINE app.

[0938] Step 6:

[0939] User interaction

[0940] Input: A message from the user.

[0941] Output: The response message generated by the generative AI model.

[0942] Specific operation: A user sends a message to their LINE account. The device sends the message to the LINE server, which passes it on to the server's chatbot. The generative AI model analyzes the message, generates an appropriate response, and sends it back to the user via the LINE API. The user receives the response in their LINE account and can continue the conversation. For example, if a user sends "Show me some recent photos of your family," the server retrieves the latest photo from the database and sends it along with the response "Here are some recent photos of your family."

[0943] (Application example 1)

[0944] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0945] In recent years, bereaved families have increasingly sought emotional support and psychological healing, but existing technologies have limited the means to provide this digitally. Furthermore, with the spread of autonomous vehicles, how users spend their time in their cars has become an important issue. In response to this, there is a need to provide a safe means for them to communicate with their deceased family members.

[0946] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0947] In this invention, the server includes: means for collecting data such as emails, images, videos, voice messages, social network logs, and letters from the electronic devices of the deceased family member; means for converting the collected data into text format and preprocessing it; means for training a generative AI model based on the preprocessed data; means for building a chatbot using the trained generative AI model; means for integrating the built chatbot into a communication interface; and means, mounted on the smart glasses, for recognizing voice input and sending user dialogue commands to the chatbot. This allows bereaved family members to experience an emotional connection with their deceased family member through dialogue with them in an autonomous vehicle and safely find healing.

[0948] "Electronic device" is a general term for devices that generate, store, transmit, or receive digital data.

[0949] "Data" refers to information such as emails, images, videos, voice messages, social network logs, letters, etc.

[0950] "Means of collection" refers to the process for extracting specific data from electronic devices and transferring it to a server, etc.

[0951] "Preprocessing means" refers to the process of converting collected data into a unified text format and organizing it in a way that makes it easier to analyze.

[0952] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses to user input based on data.

[0953] "Training" refers to the process of using preprocessed data to train a generative AI model to learn specific patterns.

[0954] A "chatbot" is a program that can understand natural language and respond to users through dialogue.

[0955] "Means of integration into a communication interface" means connecting the chatbot to a specific communication platform (e.g., LINE) and enabling it to send and receive messages.

[0956] "Smart glasses" are wearable devices that have built-in displays and sensors, display information to the user, and can be operated by voice input, touch input, etc.

[0957] "Means for recognizing voice input and transmitting user interaction commands to the chatbot" refers to the process of using a microphone and software built into the smart glasses to convert voice input into text and transmit that text to the chatbot.

[0958] An "autonomous vehicle" is a vehicle that uses sensors and AI to drive autonomously without the need for driver operation.

[0959] The system for implementing this invention aims to safely recreate conversations with deceased family members. Specifically, it collects data from the deceased family member's electronic devices, builds a chatbot using a generative AI model based on that data, and enables the user to interact with the chatbot through smart glasses.

[0960] Server-side processing

[0961] 1. Data Collection

[0962] The server collects the digital information of deceased family members (e.g., emails, images, videos, audio messages, social network logs, letters, etc.), allowing for a complete picture of their digital lives.

[0963] 2. Data Preprocessing

[0964] The collected data is preprocessed on the server. Specifically, text data is normalized, and text is extracted from images and videos using optical character recognition (OCR). Voice messages are then converted to text using speech recognition. This preprocessing process converts data from a variety of formats into a unified text format.

[0965] 3. Model Training

[0966] The preprocessed data is sent to a server as a training dataset for a generative AI model (e.g., GPT-3, BERT). The generative AI model learns from this dataset and is tuned to generate responses that mimic the language and thought patterns of the deceased family member.

[0967] 4. Chatbot construction

[0968] Using the trained generative AI model, the server builds a chatbot that can respond appropriately to messages from users.

[0969] 5. Communication interface integration

[0970] The constructed chatbot is integrated into a communication interface (e.g., LINE API), through which the server manages communication between the chatbot and the user.

[0971] User processing

[0972] 1. Voice input

[0973] Suppose a user is riding in a self-driving vehicle and wearing smart glasses. When the user speaks, "Mom, what's the weather like today?", the smart glasses recognize this as voice input. The smart glasses have a built-in microphone that captures this voice input.

[0974] 2. Voice Recognition

[0975] Software installed on the smart glasses (e.g., voice recognition software using Google Cloud API) converts this voice input into text, allowing the user's commands to be captured in text format.

[0976] 3. Send a message

[0977] The captured text is sent from the smart glasses via a communication interface to a server, where a generative AI model analyzes the user's message and generates an appropriate response.

[0978] 4. Receiving a Response

[0979] The generated response is then sent back to the smart glasses via the communication interface and presented to the user in voice or text format, for example, "The weather is sunny today, it looks like it's going to be a nice day."

[0980] Specific examples

[0981] As a concrete example, consider the case where a user in a self-driving vehicle asks, "Mom, how's the weather today?" The smart glasses recognize the speech, convert it into text, and then send it to the server. The server's generative AI model analyzes the message and generates a response to the user: "The weather is sunny today. It looks like it's going to be a nice day."

[0982] Prompt Sentence Examples

[0983] Speech prompt: "Mom, how's the weather today?"

[0984] Chatbot prompt: "A 24-hour conversational chatbot that responds based on digital information about deceased family members."

[0985] This allows users to safely interact with deceased family members from within their self-driving vehicles and receive emotional support.

[0986] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0987] Step 1:

[0988] The server collects data from the deceased family member's electronic devices, including emails, images, videos, audio messages, social network logs, and letters. The data is transferred using APIs and file transfer protocols. The input is various digital data, and the output is a unified data set stored on the server.

[0989] Step 2:

[0990] The server preprocesses the collected data by normalizing the text data, extracting text from images and videos using optical character recognition (OCR), and converting voice messages into text using speech recognition software. The input is the collected data, and the output is normalized text data.

[0991] Step 3:

[0992] The server trains a generative AI model based on the preprocessed data. Using the preprocessed data as a training dataset, an AI algorithm (e.g., GPT-3, BERT) learns language and thought patterns. The input is normalized text data, and the output is a trained generative AI model.

[0993] Step 4:

[0994] The server builds a chatbot using a trained generative AI model, which enables the AI ​​model to generate natural and appropriate responses to user input. The input is the trained generative AI model, and the output is the chatbot.

[0995] Step 5:

[0996] The server integrates the constructed chatbot into a communication interface (e.g., LINE API), allowing users to interact with the chatbot through their LINE accounts. The input is connection information between the chatbot and the communication interface, and the output is the established communication state.

[0997] Step 6:

[0998] A user gets into an autonomous vehicle and puts on smart glasses. The user speaks to the vehicle, saying, "Mom, what's the weather like today?" The input is the user's voice command, and the output is the audio data of the voice command.

[0999] Step 7:

[1000] The smart glasses use a built-in microphone to recognize the user's voice and capture acoustic data, then use speech recognition software to convert the speech into text, with the input being acoustic data and the output being text data.

[1001] Step 8:

[1002] The smart glasses send the acquired text data to the server through the communication interface. The input is the text data and the output is the transmitted text message.

[1003] Step 9:

[1004] The server uses a generative AI model to analyze the received text message and generate an appropriate response. The input is the text message from the user, and the output is the generated response text.

[1005] Step 10:

[1006] The server sends the generated response to the smart glasses through the communication interface, where the input is the generated response text and the output is the text or audio displayed on the smart glasses.

[1007] Step 11:

[1008] The smart glasses provide the received response to the user in voice or text format. The input is the transmitted response text, and the output is the information presented to the user. Specifically, the response may be something like, "The weather is sunny today, it looks like it's going to be a nice day."

[1009] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1010] This invention combines an emotion engine with a system that learns the digital information of deceased family members and recreates them as chatbots that can be interacted with 24 hours a day via LINE accounts. The system is composed of elements that play the roles of a server, a device, and a user.

[1011] System Configuration

[1012] Server-side processing

[1013] 1. Data Collection

[1014] The server collects data such as emails, photos, videos, voice messages, social networking service logs, and letters from the deceased family members' electronic devices. This involves using the device's API to obtain the data and transferring it to the server. The collected data is then stored in a database and managed centrally.

[1015] 2. Data Preprocessing

[1016] The server preprocesses the collected data. Specifically, it normalizes text data (e.g., emails, social media logs) and removes punctuation and special characters. It also performs optical character recognition (OCR) on images and videos to extract text data. Voice messages are converted to text using speech recognition. This ensures that all data is converted into a unified text format.

[1017] 3. Model Training

[1018] The server trains a generative AI model based on the preprocessed text data. Specifically, the text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[1019] 4. Chatbot construction

[1020] The server uses a trained generative AI model to build a chatbot. The chatbot receives input from the user and generates an appropriate response. The server also combines an emotion engine to recognize the user's emotions and provide a response according to the user's emotions.

[1021] 5. LINE account integration

[1022] The server integrates the chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. This integration allows users to converse with the chatbot through their LINE account.

[1023] User processing

[1024] When a user sends a message to the chatbot for a deceased family member through their LINE account, the message is sent to the server. The server's generative AI model analyzes the message and generates an appropriate response. The server's emotion engine recognizes the user's emotion from the message content and generates a response based on that emotion. The generated response is then sent back to the user via the LINE API.

[1025] Specific use cases

[1026] For example, if a user sends "I'm tired today" to the chatbot through their LINE account, the following will happen:

[1027] 1. A user sends a message through their LINE account.

[1028] Message: "I'm tired today."

[1029] 2. The device sends the message to the LINE server.

[1030] 3. The LINE server passes the message to the chatbot on the server.

[1031] 4. The server's AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the emotion "tired" and generates a response that includes words of comfort for fatigue.

[1032] Example response: "Good work. Have a good rest today."

[1033] 5. The server generates a response and sends it back to the LINE server, which then sends it to the user's LINE account.

[1034] 6. The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[1035] In this way, users can recreate the interaction experience with their deceased family members and, with the help of the emotion engine, receive more advanced emotional support, providing a means for bereaved families to receive better emotional support and cherish the memories of their deceased family members.

[1036] The processing flow will be explained below.

[1037] Step 1:

[1038] The server collects data such as emails, photos, videos, voice messages, social networking service logs, and letters from the electronic devices of deceased family members. Specifically, it retrieves and aggregates various digital data stored on the electronic devices using APIs and file transfer protocols.

[1039] Step 2:

[1040] The server pre-processes the collected data, which includes:

[1041] Normalize text data (e.g., emails and social media logs) to remove punctuation and special characters.

[1042] Optical character recognition (OCR) is used on image and video data to extract text information.

[1043] Voice messages are converted into text using voice recognition technology, which ensures that all data is converted into a unified text format.

[1044] Step 3:

[1045] The server trains a generative AI model based on the preprocessed text data, using a large dataset to train the model to reflect the deceased family member's vocabulary and thought patterns, enabling the AI ​​model to generate appropriate natural language responses.

[1046] Step 4:

[1047] The server uses the trained generative AI model to build a chatbot, which receives messages from users and generates appropriate responses.

[1048] Step 5:

[1049] The server recognizes the user's emotions using an emotion engine. Specifically, the emotion engine analyzes the input message and identifies the user's emotions (e.g., joy, sadness, anger, fatigue, etc.) using an emotion analysis algorithm.

[1050] Step 6:

[1051] Based on the analysis results of the emotion engine, the server uses a generative AI model to generate a response based on the emotion. This response is tailored to the user's current emotional state. For example, if a user sends a message saying, "I'm tired today," the emotion engine will recognize the emotion "fatigue," and the generative AI model will generate a comforting message such as, "Thank you for your hard work. Take a good rest today."

[1052] Step 7:

[1053] The server integrates the built chatbot into the communication interface. Specifically, it uses the LINE Messaging API to link the chatbot to a LINE account. This integration allows users to converse with the chatbot through their LINE account.

[1054] Step 8:

[1055] The user sends a message to the chatbot of their deceased family member through their LINE account. At this time, the user's message is sent to the server via the LINE server.

[1056] Step 9:

[1057] The server analyzes the message received from the LINE server using the generative AI model and generates an appropriate response. This includes the process where the emotion engine recognizes the user's emotions and generates a response that corresponds to those emotions.

[1058] Step 10:

[1059] The server sends the generated response back to the LINE server, which then sends it to the user's LINE account.

[1060] Step 11:

[1061] The user receives a response from the server on their LINE account and checks its contents. For example, they may receive a message such as "Thank you for your hard work. Please take a good rest today," providing emotional support.

[1062] This series of steps allows users to re-create conversations with deceased family members and, with the help of an emotion engine, receive more personalized emotional support.

[1063] Example 2

[1064] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1065] With limited means to store and relive memories of deceased family members, bereaved families find it difficult to receive emotional support or comfort through interactive experiences. To solve this problem, an advanced dialogue system that utilizes the digital information of deceased family members and also provides emotional support is needed.

[1066] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1067] In this invention, the server includes means for collecting data such as emails, images, videos, voice communications, social network service records, and letters from the information devices of the deceased family member, means for converting the collected data into text format and performing preprocessing, means for training a generative AI model based on the preprocessed data, means for constructing a dialogue system using the trained generative AI model, means for integrating the constructed dialogue system into a communication interface, emotion engine means for analyzing messages from users and recognizing their emotions, and means for generating responses based on the recognition results of the emotion engine. This allows the bereaved family to relive conversations with their deceased family member and receive support tailored to their emotions.

[1068] "Information equipment" is a general term for devices that can generate, store, manage, and transmit data such as emails, images, videos, voice communications, social network service records, and letters.

[1069] "Data collection means" refers to the functions and processes that automatically or manually acquire data from various information devices and store it on a central server.

[1070] "Data preprocessing measures" are processes such as normalization, removal of punctuation and special characters, optical character recognition (OCR) and speech recognition to convert collected data into a unified text format.

[1071] A "generative AI model" is an algorithm or artificial intelligence (AI) system that is trained based on collected text data to understand specific patterns and contexts and generate appropriate responses.

[1072] A "dialogue system" is a software application built using generative AI models to engage in dialogue with users and provide appropriate responses to their questions and comments.

[1073] A "communication interface" is a mechanism for connecting a dialogue system with a user, and often uses messaging platforms such as LINE or email.

[1074] "Emotion engine means" refers to algorithms and technologies for analyzing messages from users and recognizing the emotions contained in the content.

[1075] "Response generation means" refers to a process or function for generating an appropriate response message based on the emotion recognized by the emotion engine.

[1076] "Preprocessed data" refers to raw data collected that has been converted into text format and processed to a state suitable for normalization and analysis.

[1077] This invention relates to a system for building chatbots that can converse in real time based on the digital information of deceased family members. This system includes elements of a server, a terminal, and a user, and by combining an emotion engine, provides more advanced emotional support.

[1078] First, the server collects data from the deceased family member's electronic devices. Specifically, the hardware used utilizes data collection APIs (e.g., Gmail API, Google Photos API) to obtain data such as emails, photos, videos, voice messages, social networking service records, and letters. This data is then stored in a database on the server and managed centrally.

[1079] Next, the server preprocesses the collected data. The collected data is not suitable for analysis or training generative AI models as it is, so it must be converted into text format. Text data is normalized and punctuation and special characters are removed, and image and video data undergoes optical character recognition (OCR). Audio data is then converted into text using speech recognition technology (e.g., Google Speech-to-Text API). This preprocessing ensures that all data is converted into a unified text format.

[1080] The server then trains a generative AI model using the preprocessed text data. The generative AI model uses a natural language processing library (e.g., Transformers). This process involves providing the preprocessed data as input to the AI ​​model, training it to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[1081] Once the learning is complete, the server uses the trained generative AI model to build a chatbot. The chatbot receives messages from users and generates appropriate responses. Furthermore, by incorporating an emotion engine (e.g., Sentiment Analysis) to recognize the user's emotions, the chatbot provides responses that correspond to the user's emotions.

[1082] The server then integrates the chatbot into the communication interface. This process involves linking the chatbot to a LINE account using the LINE Messaging API. This integration allows users to interact with the chatbot through the LINE platform.

[1083] When a user sends a message through their LINE account, the message is sent via their device to the LINE server, where it is then forwarded to a chatbot on the server. The generative AI model on the server analyzes the user's message and generates an appropriate response. The emotion engine recognizes the user's emotion from the content of the message and generates a response based on that emotion. The generated response is then sent back to the user via the LINE API.

[1084] Specific examples

[1085] For example, if a user sends "I'm tired today" to a chatbot through their LINE account, the process will be as follows:

[1086] 1. A user sends a message saying, "I'm tired today."

[1087] 2. The device sends the message to the LINE server.

[1088] 3. The LINE server forwards the message to the chatbot on the server.

[1089] 4. The server's AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the emotion "tired" and generates a response that includes words of comfort for fatigue.

[1090] Example response: "Good work. Have a good rest today."

[1091] 5. The server generates a response and sends it back to the LINE server.

[1092] 6. The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[1093] In this way, users can recreate the interaction experience with their deceased family members and, with the help of the emotion engine, receive higher levels of emotional support.

[1094] Prompt Sentence Examples

[1095] Here are some examples of input prompts for generative AI models:

[1096] "I've been so busy and tired today. Could you please give me some words of encouragement?"

[1097] "I've been feeling a bit down lately...what should I do?"

[1098] "I'm really looking forward to talking to you."

[1099] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1100] Step 1:

[1101] Data collection

[1102] The server collects data such as emails, images, videos, voice communications, social networking service records, and letters from the information devices of deceased family members.

[1103] Specifically, it uses the Gmail API to collect emails, the Google Photos API to download photos and videos, and retrieves voice messages from the device's storage.

[1104] Input: Data retrieval request from information device.

[1105] Output: Raw data (emails, images, videos, audio messages, etc.) stored in a server database.

[1106] Step 2:

[1107] Data Preprocessing

[1108] The server preprocesses the collected raw data.

[1109] Text data is normalized to remove punctuation and special characters, image and video data is converted to text using OCR (Optical Character Recognition), and voice messages are converted to text using speech recognition technology.

[1110] As a concrete example, we use the Google Speech-to-Text API to convert voice messages into text and Tesseract OCR to extract text from images.

[1111] Input: Raw data stored in the server database.

[1112] Output: Preprocessed text data.

[1113] Step 3:

[1114] Model learning

[1115] The server uses the preprocessed text data to train a generative AI model.

[1116] Use a natural language processing library (e.g., Transformers) to train your AI model using preprocessed data.

[1117] Specifically, the collected data sets are batch processed and input into the AI ​​model sequentially.

[1118] Input: Preprocessed text data.

[1119] Output: A trained generative AI model.

[1120] Step 4:

[1121] Chatbot construction

[1122] The server builds a chatbot using a trained generative AI model.

[1123] Chatbots have the ability to analyze messages from users and generate appropriate responses. Furthermore, by incorporating an emotion engine (e.g., Sentiment Analysis), they can recognize the user's emotions and provide responses that correspond to those emotions.

[1124] Specifically, it incorporates a sentiment analysis library to add emotional elements to the generated responses.

[1125] Input: Trained generative AI model and emotion engine.

[1126] Output: Available chatbot systems.

[1127] Step 5:

[1128] LINE account integration

[1129] The server integrates the built chatbot into the communication interface.

[1130] Use the LINE Messaging API to link your chatbot to your LINE account, allowing users to interact with your chatbot through the LINE Platform.

[1131] Specifically, the API endpoint is set using the channel access token obtained from the LINE Developer Console and registered in the chatbot's message receiving mechanism.

[1132] Input: Chatbot system and LINE Messaging API.

[1133] Output: Chatbot linked to LINE account.

[1134] Step 6:

[1135] Sending a message and generating a response

[1136] When a user sends a message through their LINE account, the message is sent via the device to the LINE server, and then forwarded to a chatbot on the server.

[1137] The server's generative AI model analyzes the user's message and generates an appropriate response. The emotion engine recognizes the user's emotion from the message content and generates a response based on that emotion.

[1138] Example: When a user sends "I'm tired today," the emotion engine recognizes the emotion "tired" and generates a response like "Great work. Take a good rest today."

[1139] Input: A message from the user.

[1140] Output: An appropriate response message that takes emotion into consideration.

[1141] Step 7:

[1142] Returning a response

[1143] The server generates a response message and sends it back to the LINE server, which then sends it to the user's LINE account.

[1144] The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[1145] Specifically, the generated response message is sent as an API request to the LINE Messaging API and delivered to the user's LINE account.

[1146] Input: The server-generated response message.

[1147] Output: A response message that will be displayed on the user's LINE account.

[1148] (Application example 2)

[1149] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1150] In today's world, the means for remembering the deceased are limited, making it difficult to maintain emotional connections. Furthermore, the means for identifying and purchasing products and mementos related to the deceased are not highly personalized. As a result, it is difficult for bereaved families to experience a memorial experience that is emotionally supported. This invention aims to solve this problem by providing a system that learns the digital information of the deceased, provides emotional support through dialogue with the user, and recommends products related to the deceased.

[1151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1152] In this invention, the server includes: means for collecting data such as electronic messages, electronic images, electronic videos, audio messages, social network service logs, and letters from the electronic devices of the deceased family member; means for converting the collected data into text format and preprocessing it; means for training a generative AI model based on the preprocessed data; means for building a chatbot using the trained generative AI model; means for integrating the built chatbot into a communication interface; and means for providing personalized product recommendations through the user interface. This allows the user to maintain an emotional connection with the deceased through conversations with the deceased, receive recommendations for products related to the deceased, and enjoy a special shopping experience.

[1153] An "electronic message" is text information sent or received in digital form, such as an email or social media message.

[1154] "Electronic image" refers to a still image or photograph stored in digital format.

[1155] "Electronic moving images" refers to moving images or video clips stored in digital format.

[1156] A "voice message" is a message that includes recorded voice data.

[1157] "Social Network Service Logs" refers to records of user activities and interactions on social networking services.

[1158] A "letter" refers to a written communication, either on paper or digitally.

[1159] A "text format" is a format that describes information using letters and symbols.

[1160] "Preprocessing" refers to the process of converting and organizing data into a form suitable for further processing and analysis.

[1161] A "generative AI model" refers to an artificial intelligence model that learns from large amounts of data and responds to or generates new data.

[1162] A "chatbot" is an automated response system for interacting with users.

[1163] A "communication interface" refers to a standardized connection method or protocol that allows different systems or devices to communicate.

[1164] "User interface" refers to the means and design by which a user interacts with a system.

[1165] "Personalized product recommendations" is a feature that suggests specific products based on a user's specific preferences and history.

[1166] This invention combines an emotion engine with a system that learns the digital information of deceased family members and recreates them as chatbots that can be interacted with 24 hours a day through a LINE account. The system is composed of a server, a device, and a user.

[1167] System Configuration

[1168] Server-side processing

[1169] 1. Data Collection

[1170] The server collects data such as electronic messages, electronic images, electronic videos, audio messages, social networking service logs, and letters from the deceased family members' electronic devices. This collection process involves using the API of the electronic devices to obtain the data and transferring it to the server. The collected data is then stored in a database and managed centrally.

[1171] 2. Data Preprocessing

[1172] The server preprocesses the collected data. Specifically, it normalizes text data (e.g., electronic messages, social media logs) and removes punctuation and special characters. It also performs optical character recognition (OCR) on electronic images and videos to extract text data. It also converts voice messages into text using speech recognition. This ensures that all data is converted into a unified text format.

[1173] 3. Model Training

[1174] The server trains a generative AI model based on the preprocessed text data. Specifically, the text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[1175] 4. Chatbot construction

[1176] The server uses a trained generative AI model to build a chatbot. The chatbot receives input from the user and generates an appropriate response. It also combines an emotion engine to recognize the user's emotions and provide a response that corresponds to the user's emotions.

[1177] 5. LINE account integration

[1178] The server integrates the chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. This integration allows users to converse with the chatbot through their LINE account.

[1179] 6. Product recommendation

[1180] The server provides personalized product recommendations through a user interface, identifying products related to the deceased through dialogue with the user and making appropriate recommendations.

[1181] User processing

[1182] When a user sends a message to the chatbot for a deceased family member through their LINE account, the message is sent to the server. The server's generative AI model analyzes the message and generates an appropriate response. The server's emotion engine also recognizes the user's emotions from the content of the message and generates a response based on those emotions. The generated response is then sent back to the user via the LINE API. At the same time, the system recommends products related to the deceased based on the content of the conversation.

[1183] Specific examples

[1184] For example, if a user asks the chatbot through their LINE account, "Today I want to make my mom's favorite dish. What would you like?", the following will happen:

[1185] 1. A user sends a message through their LINE account.

[1186] Message: "Today I want to make your mom's favorite dish. What would you like?"

[1187] 2. The device sends the message to the LINE server.

[1188] 3. The LINE server passes the message to the chatbot on the server.

[1189] 4. The server's generative AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the user's intention and generates a response based on information about the deceased's favorite dishes.

[1190] Sample response: "Mom loved spaghetti bolognese. Why don't you try making that?"

[1191] 5. The server generates a response and sends it back to the LINE server, which then sends it to the user's LINE account.

[1192] 6. The user receives a response from the server on their LINE account and can learn about their mother's favorite dishes. The chatbot will also continue to recommend related recipes and foods.

[1193] In this way, users can recreate the interactive experience of their deceased family members and enjoy a special shopping experience thanks to the emotion engine and chatbot functions.

[1194] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1195] Step 1:

[1196] The server collects data such as electronic messages, electronic images, electronic videos, audio messages, social network service logs, and letters from the electronic devices of deceased family members. As input, it uses the device's API to acquire this data and saves it in the server's database. As output, the collected data is stored in the database.

[1197] Step 2:

[1198] The server preprocesses the collected data. As input, it takes the raw data stored in the database. For text data, it normalizes it and removes punctuation and special characters. For electronic images and videos, it performs optical character recognition (OCR) to extract text data. For voice messages, it converts them to text using speech recognition. As output, it obtains data in a unified text format.

[1199] Step 3:

[1200] The server trains a generative AI model based on the preprocessed text data. The preprocessed text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses based on the data that reflect the deceased's language and thought patterns. The output is a trained generative AI model.

[1201] Step 4:

[1202] The server uses a trained generative AI model to build a chatbot. The trained generative AI model is used as input. The chatbot receives input from the user and generates an appropriate response. The output is a chatbot that can converse.

[1203] Step 5:

[1204] The server integrates the built chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. The input is the LINE Messaging API configuration information and the chatbot integration information. The output is that the user can converse with the chatbot through their LINE account.

[1205] Step 6:

[1206] The server makes product recommendations through dialogue with the user. Inputs include messages from the user and historical dialogue data. A generative AI model analyzes the user's messages and generates appropriate responses. An emotion engine recognizes the user's emotions and recommends products related to the deceased based on those emotions. The output is a personalized product recommendation message provided to the user via their LINE account.

[1207] Step 7:

[1208] The server receives feedback from users and records the feedback data to improve the system's performance. As input, it aggregates users' interaction ratings with the chatbot and their satisfaction with product recommendations. As output, the accuracy of the generative AI model and the recommendation system is improved based on the feedback.

[1209] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1210] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1211] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1212] [Fourth embodiment]

[1213] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1214] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1215] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1216] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1217] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1218] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1219] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1220] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1221] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1222] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1223] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1224] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1225] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1226] This invention relates to a system that learns the digital information of deceased family members and recreates it as a chatbot that can be interacted with 24 hours a day via a LINE account. This system is composed of elements that play the roles of a server, a terminal, and a user.

[1227] System Configuration

[1228] Server-side processing

[1229] 1. Data Collection

[1230] The server collects data from the deceased family member's electronic devices, including emails, photos, videos, voice messages, social networking logs, and letters. This data is organized into the deceased family member's digital legacy.

[1231] 2. Data Preprocessing

[1232] The collected data is preprocessed on the server. Specifically, text data is normalized, text is extracted from images and videos using optical character recognition (OCR), and voice messages are converted to text using speech recognition. This unifies the data from various formats into a unified text format.

[1233] 3. Model Training

[1234] The pre-processed data is sent to a server as a training dataset for a generative AI model, which is then trained to generate responses that mimic the language and thought patterns of the deceased family member.

[1235] 4. Chatbot construction

[1236] Using the trained generative AI model, the server builds a chatbot that responds appropriately to messages from users.

[1237] 5. LINE account integration

[1238] The chatbot is integrated into a LINE account through a communication interface. The server uses the LINE Messaging API to manage communication between the chatbot and the LINE platform.

[1239] User processing

[1240] When a user sends a message to the chatbot of a deceased family member through their LINE account, the message is sent to the server, where the generative AI model analyzes the message and generates an appropriate response, which is then sent back to the user via the LINE API.

[1241] Specific use cases

[1242] For example, consider the case where a user sends a message to a chatbot via their LINE account saying, "Show me some recent photos of my family."

[1243] 1. A user sends a message through their LINE account.

[1244] Message: "Show me a recent photo of your family."

[1245] 2. The device sends the message to the LINE server.

[1246] 3. The LINE server passes the message to the chatbot on the server.

[1247] 4. The server's generative AI model analyzes the user's message and generates an appropriate response.

[1248] Example response: "Here's a recent photo of my family." with a link or image file showing the photo

[1249] 5. The server sends the generated response through the LINE server.

[1250] 6. The user receives a response from the server in their LINE account and can view recent photos of their family.

[1251] This allows users to recreate conversations with their deceased family members, providing an experience that maintains an emotional connection. The system provides a way for bereaved families to receive emotional support while keeping the memory of their deceased family members a part of their lives.

[1252] The processing flow will be explained below.

[1253] Step 1:

[1254] The server collects data from the deceased family member's electronic devices, including emails, photos, videos, voice messages, social networking service logs, and letters. Specifically, it uses the API provided by the device to download the data and convert it into an appropriate data format. This allows various data to be aggregated in one place.

[1255] Step 2:

[1256] The server pre-processes the collected data, which includes the following steps:

[1257] Normalization of text data (emails, SNS logs), specifically removing punctuation, adjusting spaces, expanding slang and abbreviations, etc.

[1258] Perform optical character recognition (OCR) on image and video data to extract text.

[1259] Voice messages are converted into text using voice recognition technology.

[1260] Step 3:

[1261] The server trains a generative AI model based on the preprocessed text data. Specifically, the collected text data is tokenized and provided as input to the generative AI model. The model is tuned to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[1262] Step 4:

[1263] The server uses the trained generative AI model to build a chatbot. The built chatbot has the ability to receive input from the user and generate an appropriate response. The actual response that the chatbot generates depends on what the AI ​​model has learned.

[1264] Step 5:

[1265] The server integrates the chatbot into the communication interface. Specifically, it uses the LINE Messaging API to link the chatbot to a specific LINE account. This integration allows users to converse with the chatbot through their LINE account.

[1266] Step 6:

[1267] The user sends a message to the chatbot through their LINE account.

[1268] For example, if a user sends a message saying, "Show me the last photo of your family you took," the message is immediately transmitted to the LINE server.

[1269] Step 7:

[1270] The LINE server forwards the received message to the server, which then receives it and has the generative AI model analyze the message.

[1271] Step 8:

[1272] The server's generative AI model analyzes the received message and generates an appropriate response, such as "Here's the last photo of the family we took," along with a link to the photo.

[1273] Step 9:

[1274] The server sends the generated response back to the LINE server, which then sends it to the user's LINE account. The user can then view photos of the specified family member.

[1275] Through this series of steps, users can recreate the interaction experience with their deceased family members and maintain an emotional connection. The system provides a means for bereaved families to receive emotional support while maintaining a record of their deceased family members as part of their lives.

[1276] Example 1

[1277] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1278] In modern society, there is a lack of ways to continue to recreate memories and emotional connections with deceased family members. Existing technologies have difficulty integrating the diverse digital information of deceased family members to recreate a consistent interactive experience. Therefore, there is a need for a system that allows bereaved family members to receive emotional support while continuing to cherish memories of their deceased family members.

[1279] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1280] In this invention, the server includes means for collecting information such as emails, image data, video data, audio data, social network service logs, and document data from the information devices of the deceased family member, means for converting the collected data into text format and preprocessing it, means for training an artificial intelligence model generated based on the preprocessed data, means for constructing a dialogue agent using the trained generative artificial intelligence model, means for integrating the constructed dialogue agent into a communication interface, means for generating information based on the collected data and linking the dialogue agent to an external system, and means for analyzing the information and generating responses to inquiries from users. This makes it possible to recreate an emotional connection through natural dialogue using the digital information of the deceased family member, allowing the bereaved family to receive emotional support.

[1281] An "information device" is a device that has the function of generating, saving, and transferring email, image data, video data, audio data, social network service logs, document data, and the like.

[1282] A "data collection means" is a function or process for collecting specific digital data from information devices and transferring it to a server.

[1283] "Preprocessing means" refers to a function or process that analyzes collected digital data and converts the data into a text format for uniform formatting.

[1284] A "generated artificial intelligence model" is an AI technology that has the ability to learn from a specific dataset and generate responses in natural language.

[1285] A "dialogue agent" is a program or system that has the ability to respond to messages from a user using a generated artificial intelligence model.

[1286] A "communication interface" is a protocol or mechanism for sending and receiving information between an interactive agent and an external messaging platform.

[1287] "External system integration means" refers to a function that enables a conversational agent to operate in conjunction with other systems or platforms (e.g., messaging apps).

[1288] The "information analysis means" is a function or process for analyzing messages from users and generating optimal response content.

[1289] The "response generation means" is a function or process for generating an appropriate response to be provided to the user based on the analysis results.

[1290] This invention relates to a system that trains a generative AI model based on the digital information of deceased family members and recreates it as a chatbot that can interact with them 24 hours a day. Specific embodiments of this system are described below.

[1291] Data collection

[1292] The server collects various information from the information devices of deceased family members, including emails, image data, video data, audio data, social network service logs, and document data. Dedicated scripts and programs are used in the collection process, and this data is transferred to the server. Specifically, this occurs when a user uploads a photo folder on their smartphone using a dedicated application.

[1293] Data Preprocessing

[1294] The server analyzes the collected data and converts it into a text format to standardize it. It uses optical character recognition (OCR) technologies, such as Google Cloud Vision API and Tesseract, to normalize the text data and process image and video data. It also converts voice messages into text using speech recognition technologies, such as Google Speech-to-Text API. This ensures that all data is in a unified text format.

[1295] Model learning

[1296] The server uses the preprocessed data to train generative AI models, such as OpenAI's GPT-3 and ChatGPT models, which are trained to generate natural-sounding responses based on the deceased family member's vocabulary and thought patterns based on the text data.

[1297] Chatbot construction

[1298] The server uses the trained generative AI model to build a conversational agent, which is implemented as an API using the Python Flask framework, resulting in a chatbot that responds to messages from users.

[1299] LINE account integration

[1300] The server integrates the built conversational agent with a LINE account. Specifically, it uses the LINE Messaging API to manage communication between the chatbot and the LINE platform. When a user sends a message via LINE, the message is forwarded to the server, where the generative AI model analyzes it and generates an appropriate response. The generated response is then sent back to the user via the LINE API.

[1301] Examples of concrete examples and prompts

[1302] For example, here is a specific example of a user sending a request to a chatbot via their LINE account saying, "Show me recent photos of my family."

[1303] 1. A user sends a message through their LINE account.

[1304] Message: "Show me a recent photo of your family."

[1305] 2. The server's AI model analyzes this message and generates an appropriate response.

[1306] Example response: "Here's a recent photo of my family." with a link or image file showing the photo

[1307] Thus, a specific example of a prompt that a generative AI model can use to generate an appropriate response is "Hello, can you show me a recent family photo?" Because the model is trained on this prompt, it can provide a natural response to the user.

[1308] In this way, the bereaved can relive the conversations with their deceased loved ones and maintain emotional ties, which is a specific embodiment of the present invention.

[1309] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1310] Step 1:

[1311] Data collection

[1312] Input: Emails, image data, video data, audio data, social networking service logs, document data, etc. from the information devices of deceased family members.

[1313] Output: The collection of digital data transferred to the server.

[1314] Specific operation: The user uploads data from their smartphone or PC using a dedicated application. The data is collected by a dedicated script and transferred to the server. For example, the user selects a photo folder on their smartphone and presses the upload button, which sends the data to the server.

[1315] Step 2:

[1316] Data Preprocessing

[1317] Input: A collection of digital data transferred to the server.

[1318] Output: A normalized set of data converted to text format.

[1319] What it does: The server analyzes the collected data and converts it into text format. For example, it uses the Google Cloud Vision API to extract text from images and video data, and the Google Speech-to-Text API to convert audio data into text. Furthermore, it performs spell checking and grammar correction as part of the normalization process.

[1320] Step 3:

[1321] Model learning

[1322] Input: Preprocessed data in text format.

[1323] Output: A trained generative AI model.

[1324] How it works: The server inputs the preprocessed data into a generative AI model (e.g., GPT-3) and trains the model. Based on a large amount of text data, the model learns the language and thought patterns of the deceased family member. A dedicated training script is used for this training process. For example, preprocessed text data from emails or letters is input into the model, and training proceeds over a period of several hours to several days.

[1325] Step 4:

[1326] Chatbot construction

[1327] Input: A trained generative AI model.

[1328] Output: A chatbot API that can respond to user input.

[1329] How it works: The server uses a trained generative AI model to build a conversational agent. An API server is configured using Python's Flask framework, and the generative AI model is connected to this API. This creates a mechanism for the generative AI model to respond appropriately to external requests. For example, if an endpoint called " / message" is set up on the Flask server, when a message is POSTed there, the generative AI model will generate and return a response.

[1330] Step 5:

[1331] LINE account integration

[1332] Input: A LINE message from the user.

[1333] Output: The response of the generated AI model through a LINE account.

[1334] How it works: The server integrates the conversational agent into a LINE account using the LINE Messaging API. Create a channel on the LINE Developers site and set the API key in the Flask server. When a user sends a message from the LINE app, the message is forwarded to the server, where the generative AI model analyzes it and generates a response, which is then sent back to the user via the LINE API. For example, when a user sends "Hello" on LINE, the server passes the message to the generative AI model, which generates the response "Hello, how are you?" and sends it back to the LINE app.

[1335] Step 6:

[1336] User interaction

[1337] Input: A message from the user.

[1338] Output: The response message generated by the generative AI model.

[1339] Specific operation: A user sends a message to their LINE account. The device sends the message to the LINE server, which passes it on to the server's chatbot. The generative AI model analyzes the message, generates an appropriate response, and sends it back to the user via the LINE API. The user receives the response in their LINE account and can continue the conversation. For example, if a user sends "Show me some recent photos of your family," the server retrieves the latest photo from the database and sends it along with the response "Here are some recent photos of your family."

[1340] (Application example 1)

[1341] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1342] In recent years, bereaved families have increasingly sought emotional support and psychological healing, but existing technologies have limited the means to provide this digitally. Furthermore, with the spread of autonomous vehicles, how users spend their time in their cars has become an important issue. In response to this, there is a need to provide a safe means for them to communicate with their deceased family members.

[1343] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1344] In this invention, the server includes: means for collecting data such as emails, images, videos, voice messages, social network logs, and letters from the electronic devices of the deceased family member; means for converting the collected data into text format and preprocessing it; means for training a generative AI model based on the preprocessed data; means for building a chatbot using the trained generative AI model; means for integrating the built chatbot into a communication interface; and means, mounted on the smart glasses, for recognizing voice input and sending user dialogue commands to the chatbot. This allows bereaved family members to experience an emotional connection with their deceased family member through dialogue with them in an autonomous vehicle and safely find healing.

[1345] "Electronic device" is a general term for devices that generate, store, transmit, or receive digital data.

[1346] "Data" refers to information such as emails, images, videos, voice messages, social network logs, letters, etc.

[1347] "Means of collection" refers to the process for extracting specific data from electronic devices and transferring it to a server, etc.

[1348] "Preprocessing means" refers to the process of converting collected data into a unified text format and organizing it in a way that makes it easier to analyze.

[1349] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses to user input based on data.

[1350] "Training" refers to the process of using preprocessed data to train a generative AI model to learn specific patterns.

[1351] A "chatbot" is a program that can understand natural language and respond to users through dialogue.

[1352] "Means of integration into a communication interface" means connecting the chatbot to a specific communication platform (e.g., LINE) and enabling it to send and receive messages.

[1353] "Smart glasses" are wearable devices that have built-in displays and sensors, display information to the user, and can be operated by voice input, touch input, etc.

[1354] "Means for recognizing voice input and transmitting user interaction commands to the chatbot" refers to the process of using a microphone and software built into the smart glasses to convert voice input into text and transmit that text to the chatbot.

[1355] An "autonomous vehicle" is a vehicle that uses sensors and AI to drive autonomously without the need for driver operation.

[1356] The system for implementing this invention aims to safely recreate conversations with deceased family members. Specifically, it collects data from the deceased family member's electronic devices, builds a chatbot using a generative AI model based on that data, and enables the user to interact with the chatbot through smart glasses.

[1357] Server-side processing

[1358] 1. Data Collection

[1359] The server collects the digital information of deceased family members (e.g., emails, images, videos, audio messages, social network logs, letters, etc.), allowing for a complete picture of their digital lives.

[1360] 2. Data Preprocessing

[1361] The collected data is preprocessed on the server. Specifically, text data is normalized, and text is extracted from images and videos using optical character recognition (OCR). Voice messages are then converted to text using speech recognition. This preprocessing process converts data from a variety of formats into a unified text format.

[1362] 3. Model Training

[1363] The preprocessed data is sent to a server as a training dataset for a generative AI model (e.g., GPT-3, BERT). The generative AI model learns from this dataset and is tuned to generate responses that mimic the language and thought patterns of the deceased family member.

[1364] 4. Chatbot construction

[1365] Using the trained generative AI model, the server builds a chatbot that can respond appropriately to messages from users.

[1366] 5. Communication interface integration

[1367] The constructed chatbot is integrated into a communication interface (e.g., LINE API), through which the server manages communication between the chatbot and the user.

[1368] User processing

[1369] 1. Voice input

[1370] Suppose a user is riding in a self-driving vehicle and wearing smart glasses. When the user speaks, "Mom, what's the weather like today?", the smart glasses recognize this as voice input. The smart glasses have a built-in microphone that captures this voice input.

[1371] 2. Voice Recognition

[1372] Software installed on the smart glasses (e.g., voice recognition software using Google Cloud API) converts this voice input into text, allowing the user's commands to be captured in text format.

[1373] 3. Send a message

[1374] The captured text is sent from the smart glasses via a communication interface to a server, where a generative AI model analyzes the user's message and generates an appropriate response.

[1375] 4. Receiving a Response

[1376] The generated response is then sent back to the smart glasses via the communication interface and presented to the user in voice or text format, for example, "The weather is sunny today, it looks like it's going to be a nice day."

[1377] Specific examples

[1378] As a concrete example, consider the case where a user in a self-driving vehicle asks, "Mom, how's the weather today?" The smart glasses recognize the speech, convert it into text, and then send it to the server. The server's generative AI model analyzes the message and generates a response to the user: "The weather is sunny today. It looks like it's going to be a nice day."

[1379] Prompt Sentence Examples

[1380] Speech prompt: "Mom, how's the weather today?"

[1381] Chatbot prompt: "A 24-hour conversational chatbot that responds based on digital information about deceased family members."

[1382] This allows users to safely interact with deceased family members from within their self-driving vehicles and receive emotional support.

[1383] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1384] Step 1:

[1385] The server collects data from the deceased family member's electronic devices, including emails, images, videos, audio messages, social network logs, and letters. The data is transferred using APIs and file transfer protocols. The input is various digital data, and the output is a unified data set stored on the server.

[1386] Step 2:

[1387] The server preprocesses the collected data by normalizing the text data, extracting text from images and videos using optical character recognition (OCR), and converting voice messages into text using speech recognition software. The input is the collected data, and the output is normalized text data.

[1388] Step 3:

[1389] The server trains a generative AI model based on the preprocessed data. Using the preprocessed data as a training dataset, an AI algorithm (e.g., GPT-3, BERT) learns language and thought patterns. The input is normalized text data, and the output is a trained generative AI model.

[1390] Step 4:

[1391] The server builds a chatbot using a trained generative AI model, which enables the AI ​​model to generate natural and appropriate responses to user input. The input is the trained generative AI model, and the output is the chatbot.

[1392] Step 5:

[1393] The server integrates the constructed chatbot into a communication interface (e.g., LINE API), allowing users to interact with the chatbot through their LINE accounts. The input is connection information between the chatbot and the communication interface, and the output is the established communication state.

[1394] Step 6:

[1395] A user gets into an autonomous vehicle and puts on smart glasses. The user speaks to the vehicle, saying, "Mom, what's the weather like today?" The input is the user's voice command, and the output is the audio data of the voice command.

[1396] Step 7:

[1397] The smart glasses use a built-in microphone to recognize the user's voice and capture acoustic data, then use speech recognition software to convert the speech into text, with the input being acoustic data and the output being text data.

[1398] Step 8:

[1399] The smart glasses send the acquired text data to the server through the communication interface. The input is the text data and the output is the transmitted text message.

[1400] Step 9:

[1401] The server uses a generative AI model to analyze the received text message and generate an appropriate response. The input is the text message from the user, and the output is the generated response text.

[1402] Step 10:

[1403] The server sends the generated response to the smart glasses through the communication interface, where the input is the generated response text and the output is the text or audio displayed on the smart glasses.

[1404] Step 11:

[1405] The smart glasses provide the received response to the user in voice or text format. The input is the transmitted response text, and the output is the information presented to the user. Specifically, the response may be something like, "The weather is sunny today, it looks like it's going to be a nice day."

[1406] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1407] This invention combines an emotion engine with a system that learns the digital information of deceased family members and recreates them as chatbots that can be interacted with 24 hours a day via LINE accounts. The system is composed of elements that play the roles of a server, a device, and a user.

[1408] System Configuration

[1409] Server-side processing

[1410] 1. Data Collection

[1411] The server collects data such as emails, photos, videos, voice messages, social networking service logs, and letters from the deceased family members' electronic devices. This involves using the device's API to obtain the data and transferring it to the server. The collected data is then stored in a database and managed centrally.

[1412] 2. Data Preprocessing

[1413] The server preprocesses the collected data. Specifically, it normalizes text data (e.g., emails, social media logs) and removes punctuation and special characters. It also performs optical character recognition (OCR) on images and videos to extract text data. Voice messages are converted to text using speech recognition. This ensures that all data is converted into a unified text format.

[1414] 3. Model Training

[1415] The server trains a generative AI model based on the preprocessed text data. Specifically, the text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[1416] 4. Chatbot construction

[1417] The server uses a trained generative AI model to build a chatbot. The chatbot receives input from the user and generates an appropriate response. The server also combines an emotion engine to recognize the user's emotions and provide a response according to the user's emotions.

[1418] 5. LINE account integration

[1419] The server integrates the chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. This integration allows users to converse with the chatbot through their LINE account.

[1420] User processing

[1421] When a user sends a message to the chatbot for a deceased family member through their LINE account, the message is sent to the server. The server's generative AI model analyzes the message and generates an appropriate response. The server's emotion engine recognizes the user's emotion from the message content and generates a response based on that emotion. The generated response is then sent back to the user via the LINE API.

[1422] Specific use cases

[1423] For example, if a user sends "I'm tired today" to the chatbot through their LINE account, the following will happen:

[1424] 1. A user sends a message through their LINE account.

[1425] Message: "I'm tired today."

[1426] 2. The device sends the message to the LINE server.

[1427] 3. The LINE server passes the message to the chatbot on the server.

[1428] 4. The server's AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the emotion "tired" and generates a response that includes words of comfort for fatigue.

[1429] Example response: "Good work. Have a good rest today."

[1430] 5. The server generates a response and sends it back to the LINE server, which then sends it to the user's LINE account.

[1431] 6. The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[1432] In this way, users can recreate the interaction experience with their deceased family members and, with the help of the emotion engine, receive more advanced emotional support, providing a means for bereaved families to receive better emotional support and cherish the memories of their deceased family members.

[1433] The processing flow will be explained below.

[1434] Step 1:

[1435] The server collects data such as emails, photos, videos, voice messages, social networking service logs, and letters from the electronic devices of deceased family members. Specifically, it retrieves and aggregates various digital data stored on the electronic devices using APIs and file transfer protocols.

[1436] Step 2:

[1437] The server pre-processes the collected data, which includes:

[1438] Normalize text data (e.g., emails and social media logs) to remove punctuation and special characters.

[1439] Optical character recognition (OCR) is used on image and video data to extract text information.

[1440] Voice messages are converted into text using voice recognition technology, which ensures that all data is converted into a unified text format.

[1441] Step 3:

[1442] The server trains a generative AI model based on the preprocessed text data, using a large dataset to train the model to reflect the deceased family member's vocabulary and thought patterns, enabling the AI ​​model to generate appropriate natural language responses.

[1443] Step 4:

[1444] The server uses the trained generative AI model to build a chatbot, which receives messages from users and generates appropriate responses.

[1445] Step 5:

[1446] The server recognizes the user's emotions using an emotion engine. Specifically, the emotion engine analyzes the input message and identifies the user's emotions (e.g., joy, sadness, anger, fatigue, etc.) using an emotion analysis algorithm.

[1447] Step 6:

[1448] Based on the analysis results of the emotion engine, the server uses a generative AI model to generate a response based on the emotion. This response is tailored to the user's current emotional state. For example, if a user sends a message saying, "I'm tired today," the emotion engine will recognize the emotion "fatigue," and the generative AI model will generate a comforting message such as, "Thank you for your hard work. Take a good rest today."

[1449] Step 7:

[1450] The server integrates the built chatbot into the communication interface. Specifically, it uses the LINE Messaging API to link the chatbot to a LINE account. This integration allows users to converse with the chatbot through their LINE account.

[1451] Step 8:

[1452] The user sends a message to the chatbot of their deceased family member through their LINE account. At this time, the user's message is sent to the server via the LINE server.

[1453] Step 9:

[1454] The server analyzes the message received from the LINE server using the generative AI model and generates an appropriate response. This includes the process where the emotion engine recognizes the user's emotions and generates a response that corresponds to those emotions.

[1455] Step 10:

[1456] The server sends the generated response back to the LINE server, which then sends it to the user's LINE account.

[1457] Step 11:

[1458] The user receives a response from the server on their LINE account and checks its contents. For example, they may receive a message such as "Thank you for your hard work. Please take a good rest today," providing emotional support.

[1459] This series of steps allows users to re-create conversations with deceased family members and, with the help of an emotion engine, receive more personalized emotional support.

[1460] Example 2

[1461] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1462] With limited means to store and relive memories of deceased family members, bereaved families find it difficult to receive emotional support or comfort through interactive experiences. To solve this problem, an advanced dialogue system that utilizes the digital information of deceased family members and also provides emotional support is needed.

[1463] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1464] In this invention, the server includes means for collecting data such as emails, images, videos, voice communications, social network service records, and letters from the information devices of the deceased family member, means for converting the collected data into text format and performing preprocessing, means for training a generative AI model based on the preprocessed data, means for constructing a dialogue system using the trained generative AI model, means for integrating the constructed dialogue system into a communication interface, emotion engine means for analyzing messages from users and recognizing their emotions, and means for generating responses based on the recognition results of the emotion engine. This allows the bereaved family to relive conversations with their deceased family member and receive support tailored to their emotions.

[1465] "Information equipment" is a general term for devices that can generate, store, manage, and transmit data such as emails, images, videos, voice communications, social network service records, and letters.

[1466] "Data collection means" refers to the functions and processes that automatically or manually acquire data from various information devices and store it on a central server.

[1467] "Data preprocessing measures" are processes such as normalization, removal of punctuation and special characters, optical character recognition (OCR) and speech recognition to convert collected data into a unified text format.

[1468] A "generative AI model" is an algorithm or artificial intelligence (AI) system that is trained based on collected text data to understand specific patterns and contexts and generate appropriate responses.

[1469] A "dialogue system" is a software application built using generative AI models to engage in dialogue with users and provide appropriate responses to their questions and comments.

[1470] A "communication interface" is a mechanism for connecting a dialogue system with a user, and often uses messaging platforms such as LINE or email.

[1471] "Emotion engine means" refers to algorithms and technologies for analyzing messages from users and recognizing the emotions contained in the content.

[1472] "Response generation means" refers to a process or function for generating an appropriate response message based on the emotion recognized by the emotion engine.

[1473] "Preprocessed data" refers to raw data collected that has been converted into text format and processed to a state suitable for normalization and analysis.

[1474] This invention relates to a system for building chatbots that can converse in real time based on the digital information of deceased family members. This system includes elements of a server, a terminal, and a user, and by combining an emotion engine, provides more advanced emotional support.

[1475] First, the server collects data from the deceased family member's electronic devices. Specifically, the hardware used utilizes data collection APIs (e.g., Gmail API, Google Photos API) to obtain data such as emails, photos, videos, voice messages, social networking service records, and letters. This data is then stored in a database on the server and managed centrally.

[1476] Next, the server preprocesses the collected data. The collected data is not suitable for analysis or training generative AI models as it is, so it must be converted into text format. Text data is normalized and punctuation and special characters are removed, and image and video data undergoes optical character recognition (OCR). Audio data is then converted into text using speech recognition technology (e.g., Google Speech-to-Text API). This preprocessing ensures that all data is converted into a unified text format.

[1477] The server then trains a generative AI model using the preprocessed text data. The generative AI model uses a natural language processing library (e.g., Transformers). This process involves providing the preprocessed data as input to the AI ​​model, training it to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[1478] Once the learning is complete, the server uses the trained generative AI model to build a chatbot. The chatbot receives messages from users and generates appropriate responses. Furthermore, by incorporating an emotion engine (e.g., Sentiment Analysis) to recognize the user's emotions, the chatbot provides responses that correspond to the user's emotions.

[1479] The server then integrates the chatbot into the communication interface. This process involves linking the chatbot to a LINE account using the LINE Messaging API. This integration allows users to interact with the chatbot through the LINE platform.

[1480] When a user sends a message through their LINE account, the message is sent via their device to the LINE server, where it is then forwarded to a chatbot on the server. The generative AI model on the server analyzes the user's message and generates an appropriate response. The emotion engine recognizes the user's emotion from the content of the message and generates a response based on that emotion. The generated response is then sent back to the user via the LINE API.

[1481] Specific examples

[1482] For example, if a user sends "I'm tired today" to a chatbot through their LINE account, the process will be as follows:

[1483] 1. A user sends a message saying, "I'm tired today."

[1484] 2. The device sends the message to the LINE server.

[1485] 3. The LINE server forwards the message to the chatbot on the server.

[1486] 4. The server's AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the emotion "tired" and generates a response that includes words of comfort for fatigue.

[1487] Example response: "Good work. Have a good rest today."

[1488] 5. The server generates a response and sends it back to the LINE server.

[1489] 6. The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[1490] In this way, users can recreate the interaction experience with their deceased family members and, with the help of the emotion engine, receive higher levels of emotional support.

[1491] Prompt Sentence Examples

[1492] Here are some examples of input prompts for generative AI models:

[1493] "I've been so busy and tired today. Could you please give me some words of encouragement?"

[1494] "I've been feeling a bit down lately...what should I do?"

[1495] "I'm really looking forward to talking to you."

[1496] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1497] Step 1:

[1498] Data collection

[1499] The server collects data such as emails, images, videos, voice communications, social networking service records, and letters from the information devices of deceased family members.

[1500] Specifically, it uses the Gmail API to collect emails, the Google Photos API to download photos and videos, and retrieves voice messages from the device's storage.

[1501] Input: Data retrieval request from information device.

[1502] Output: Raw data (emails, images, videos, audio messages, etc.) stored in a server database.

[1503] Step 2:

[1504] Data Preprocessing

[1505] The server preprocesses the collected raw data.

[1506] Text data is normalized to remove punctuation and special characters, image and video data is converted to text using OCR (Optical Character Recognition), and voice messages are converted to text using speech recognition technology.

[1507] As a concrete example, we use the Google Speech-to-Text API to convert voice messages into text and Tesseract OCR to extract text from images.

[1508] Input: Raw data stored in the server database.

[1509] Output: Preprocessed text data.

[1510] Step 3:

[1511] Model learning

[1512] The server uses the preprocessed text data to train a generative AI model.

[1513] Use a natural language processing library (e.g., Transformers) to train your AI model using preprocessed data.

[1514] Specifically, the collected data sets are batch processed and input into the AI ​​model sequentially.

[1515] Input: Preprocessed text data.

[1516] Output: A trained generative AI model.

[1517] Step 4:

[1518] Chatbot construction

[1519] The server builds a chatbot using a trained generative AI model.

[1520] Chatbots have the ability to analyze messages from users and generate appropriate responses. Furthermore, by incorporating an emotion engine (e.g., Sentiment Analysis), they can recognize the user's emotions and provide responses that correspond to those emotions.

[1521] Specifically, it incorporates a sentiment analysis library to add emotional elements to the generated responses.

[1522] Input: Trained generative AI model and emotion engine.

[1523] Output: Available chatbot systems.

[1524] Step 5:

[1525] LINE account integration

[1526] The server integrates the built chatbot into the communication interface.

[1527] Use the LINE Messaging API to link your chatbot to your LINE account, allowing users to interact with your chatbot through the LINE Platform.

[1528] Specifically, the API endpoint is set using the channel access token obtained from the LINE Developer Console and registered in the chatbot's message receiving mechanism.

[1529] Input: Chatbot system and LINE Messaging API.

[1530] Output: Chatbot linked to LINE account.

[1531] Step 6:

[1532] Sending a message and generating a response

[1533] When a user sends a message through their LINE account, the message is sent via the device to the LINE server, and then forwarded to a chatbot on the server.

[1534] The server's generative AI model analyzes the user's message and generates an appropriate response. The emotion engine recognizes the user's emotion from the message content and generates a response based on that emotion.

[1535] Example: When a user sends "I'm tired today," the emotion engine recognizes the emotion "tired" and generates a response like "Great work. Take a good rest today."

[1536] Input: A message from the user.

[1537] Output: An appropriate response message that takes emotion into consideration.

[1538] Step 7:

[1539] Returning a response

[1540] The server generates a response message and sends it back to the LINE server, which then sends it to the user's LINE account.

[1541] The user receives a response from the server on their LINE account and feels comforted by the words "Thank you for your hard work."

[1542] Specifically, the generated response message is sent as an API request to the LINE Messaging API and delivered to the user's LINE account.

[1543] Input: The server-generated response message.

[1544] Output: A response message that will be displayed on the user's LINE account.

[1545] (Application example 2)

[1546] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1547] In today's world, the means for remembering the deceased are limited, making it difficult to maintain emotional connections. Furthermore, the means for identifying and purchasing products and mementos related to the deceased are not highly personalized. As a result, it is difficult for bereaved families to experience a memorial experience that is emotionally supported. This invention aims to solve this problem by providing a system that learns the digital information of the deceased, provides emotional support through dialogue with the user, and recommends products related to the deceased.

[1548] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1549] In this invention, the server includes: means for collecting data such as electronic messages, electronic images, electronic videos, audio messages, social network service logs, and letters from the electronic devices of the deceased family member; means for converting the collected data into text format and preprocessing it; means for training a generative AI model based on the preprocessed data; means for building a chatbot using the trained generative AI model; means for integrating the built chatbot into a communication interface; and means for providing personalized product recommendations through the user interface. This allows the user to maintain an emotional connection with the deceased through conversations with the deceased, receive recommendations for products related to the deceased, and enjoy a special shopping experience.

[1550] An "electronic message" is text information sent or received in digital form, such as an email or social media message.

[1551] "Electronic image" refers to a still image or photograph stored in digital format.

[1552] "Electronic moving images" refers to moving images or video clips stored in digital format.

[1553] A "voice message" is a message that includes recorded voice data.

[1554] "Social Network Service Logs" refers to records of user activities and interactions on social networking services.

[1555] A "letter" refers to a written communication, either on paper or digitally.

[1556] A "text format" is a format that describes information using letters and symbols.

[1557] "Preprocessing" refers to the process of converting and organizing data into a form suitable for further processing and analysis.

[1558] A "generative AI model" refers to an artificial intelligence model that learns from large amounts of data and responds to or generates new data.

[1559] A "chatbot" is an automated response system for interacting with users.

[1560] A "communication interface" refers to a standardized connection method or protocol that allows different systems or devices to communicate.

[1561] "User interface" refers to the means and design by which a user interacts with a system.

[1562] "Personalized product recommendations" is a feature that suggests specific products based on a user's specific preferences and history.

[1563] This invention combines an emotion engine with a system that learns the digital information of deceased family members and recreates them as chatbots that can be interacted with 24 hours a day through a LINE account. The system is composed of a server, a device, and a user.

[1564] System Configuration

[1565] Server-side processing

[1566] 1. Data Collection

[1567] The server collects data such as electronic messages, electronic images, electronic videos, audio messages, social networking service logs, and letters from the deceased family members' electronic devices. This collection process involves using the API of the electronic devices to obtain the data and transferring it to the server. The collected data is then stored in a database and managed centrally.

[1568] 2. Data Preprocessing

[1569] The server preprocesses the collected data. Specifically, it normalizes text data (e.g., electronic messages, social media logs) and removes punctuation and special characters. It also performs optical character recognition (OCR) on electronic images and videos to extract text data. It also converts voice messages into text using speech recognition. This ensures that all data is converted into a unified text format.

[1570] 3. Model Training

[1571] The server trains a generative AI model based on the preprocessed text data. Specifically, the text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses that reflect the deceased family member's vocabulary and thought patterns.

[1572] 4. Chatbot construction

[1573] The server uses a trained generative AI model to build a chatbot. The chatbot receives input from the user and generates an appropriate response. It also combines an emotion engine to recognize the user's emotions and provide a response that corresponds to the user's emotions.

[1574] 5. LINE account integration

[1575] The server integrates the chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. This integration allows users to converse with the chatbot through their LINE account.

[1576] 6. Product recommendation

[1577] The server provides personalized product recommendations through a user interface, identifying products related to the deceased through dialogue with the user and making appropriate recommendations.

[1578] User processing

[1579] When a user sends a message to the chatbot for a deceased family member through their LINE account, the message is sent to the server. The server's generative AI model analyzes the message and generates an appropriate response. The server's emotion engine also recognizes the user's emotions from the content of the message and generates a response based on those emotions. The generated response is then sent back to the user via the LINE API. At the same time, the system recommends products related to the deceased based on the content of the conversation.

[1580] Specific examples

[1581] For example, if a user asks the chatbot through their LINE account, "Today I want to make my mom's favorite dish. What would you like?", the following will happen:

[1582] 1. A user sends a message through their LINE account.

[1583] Message: "Today I want to make your mom's favorite dish. What would you like?"

[1584] 2. The device sends the message to the LINE server.

[1585] 3. The LINE server passes the message to the chatbot on the server.

[1586] 4. The server's generative AI model analyzes the user's message and generates an appropriate response. Furthermore, the emotion engine recognizes the user's intention and generates a response based on information about the deceased's favorite dishes.

[1587] Sample response: "Mom loved spaghetti bolognese. Why don't you try making that?"

[1588] 5. The server generates a response and sends it back to the LINE server, which then sends it to the user's LINE account.

[1589] 6. The user receives a response from the server on their LINE account and can learn about their mother's favorite dishes. The chatbot will also continue to recommend related recipes and foods.

[1590] In this way, users can recreate the interactive experience of their deceased family members and enjoy a special shopping experience thanks to the emotion engine and chatbot functions.

[1591] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1592] Step 1:

[1593] The server collects data such as electronic messages, electronic images, electronic videos, audio messages, social network service logs, and letters from the electronic devices of deceased family members. As input, it uses the device's API to acquire this data and saves it in the server's database. As output, the collected data is stored in the database.

[1594] Step 2:

[1595] The server preprocesses the collected data. As input, it takes the raw data stored in the database. For text data, it normalizes it and removes punctuation and special characters. For electronic images and videos, it performs optical character recognition (OCR) to extract text data. For voice messages, it converts them to text using speech recognition. As output, it obtains data in a unified text format.

[1596] Step 3:

[1597] The server trains a generative AI model based on the preprocessed text data. The preprocessed text data is provided as input to the AI ​​model. The generative AI model is trained to generate responses based on the data that reflect the deceased's language and thought patterns. The output is a trained generative AI model.

[1598] Step 4:

[1599] The server uses a trained generative AI model to build a chatbot. The trained generative AI model is used as input. The chatbot receives input from the user and generates an appropriate response. The output is a chatbot that can converse.

[1600] Step 5:

[1601] The server integrates the built chatbot into the communication interface. Specifically, it links the chatbot to a LINE account using the LINE Messaging API. The input is the LINE Messaging API configuration information and the chatbot integration information. The output is that the user can converse with the chatbot through their LINE account.

[1602] Step 6:

[1603] The server makes product recommendations through dialogue with the user. Inputs include messages from the user and historical dialogue data. A generative AI model analyzes the user's messages and generates appropriate responses. An emotion engine recognizes the user's emotions and recommends products related to the deceased based on those emotions. The output is a personalized product recommendation message provided to the user via their LINE account.

[1604] Step 7:

[1605] The server receives feedback from users and records the feedback data to improve the system's performance. As input, it aggregates users' interaction ratings with the chatbot and their satisfaction with product recommendations. As output, the accuracy of the generative AI model and the recommendation system is improved based on the feedback.

[1606] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1607] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1608] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1609] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1610] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1611] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1612] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1613] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1614] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1615] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1616] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1617] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1618] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1619] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1620] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1621] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1622] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1623] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1624] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1625] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1626] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1627] The following is further disclosed regarding the above embodiment.

[1628] (Claim 1)

[1629] A method for collecting data such as emails, photos, videos, voice messages, social networking service logs, and letters from the electronic devices of deceased family members;

[1630] a means for converting the collected data into a text format and preprocessing the data;

[1631] A means of training a generative AI model based on the preprocessed data; and

[1632] A means to build a chatbot using a trained generative AI model;

[1633] A means of integrating the constructed chatbot into a communication interface;

[1634] A system including:

[1635] (Claim 2)

[1636] 10. The system of claim 1, wherein optical character recognition is performed on the preprocessed data to convert the image and video data to text.

[1637] (Claim 3)

[1638] 10. The system of claim 1, wherein speech recognition is performed on the preprocessed data and the speech data is converted to text.

[1639] "Example 1"

[1640] (Claim 1)

[1641] A means for collecting information such as emails, image data, video data, audio data, social network service logs, and document data from the information devices of the deceased family member;

[1642] a means for converting the collected data into a text format and preprocessing it;

[1643] A means for training an artificial intelligence model generated based on the preprocessed data;

[1644] A means for constructing a dialogue agent using a trained generative artificial intelligence model;

[1645] a means for integrating the constructed dialogue agent into a communication interface;

[1646] A means to generate information based on the collected data and link the dialogue agent to external systems;

[1647] means for analyzing the information and generating responses to user queries;

[1648] A system including:

[1649] (Claim 2)

[1650] 10. The system of claim 1, wherein optical character recognition is performed on the preprocessed data to convert the image and video data to text.

[1651] (Claim 3)

[1652] 10. The system of claim 1, wherein speech recognition is performed on the preprocessed data and the speech data is converted to text.

[1653] "Application Example 1"

[1654] (Claim 1)

[1655] A method to collect data such as emails, images, videos, audio messages, social network logs, and letters from the electronic devices of deceased family members;

[1656] a means for converting the collected data into a text format and preprocessing the data;

[1657] A means of training a generative AI model based on the preprocessed data; and

[1658] A means to build a chatbot using a trained generative AI model;

[1659] A means of integrating the constructed chatbot into a communication interface;

[1660] means, mounted on the smart glasses, for recognizing voice input and transmitting user dialogue commands to the chatbot;

[1661] A system including:

[1662] (Claim 2)

[1663] 10. The system of claim 1, wherein optical character recognition is performed on the preprocessed data to convert image and video data into text.

[1664] (Claim 3)

[1665] 10. The system of claim 1, wherein speech recognition is performed on the preprocessed data and the speech data is converted to text.

[1666] "Example 2: Combining Emotion Engines"

[1667] (Claim 1)

[1668] A method for collecting data such as emails, images, videos, voice communications, social networking service records, and letters from the information devices of deceased family members;

[1669] A means for converting the collected data into a text format and preprocessing it;

[1670] a means for training a generative AI model based on the preprocessed data; and

[1671] A means of building a dialogue system using a trained generative AI model;

[1672] a means for integrating the constructed dialogue system into a communication interface;

[1673] emotion engine means for analyzing messages from users and recognizing their emotions;

[1674] means for generating a response based on the recognition result by the emotion engine;

[1675] A system including:

[1676] (Claim 2)

[1677] 10. The system of claim 1, wherein optical character recognition is performed on the preprocessed data to convert image and video data into text.

[1678] (Claim 3)

[1679] 10. The system of claim 1, wherein speech recognition is performed on the preprocessed data and the speech data is converted to text.

[1680] "Application example 2 when combining emotion engines"

[1681] (Claim 1)

[1682] A method for collecting data such as electronic messages, electronic images, electronic videos, audio messages, social networking service logs, and letters from the electronic devices of deceased family members;

[1683] a means for converting the collected data into a text format and preprocessing the data;

[1684] A means of training a generative AI model based on the preprocessed data; and

[1685] A means to build a chatbot using a trained generative AI model;

[1686] A means of integrating the constructed chatbot into a communication interface;

[1687] a means for providing personalized product recommendations through a user interface;

[1688] A system including:

[1689] (Claim 2)

[1690] 10. The system of claim 1, wherein optical character recognition is performed on the preprocessed data to convert the image and video data to text.

[1691] (Claim 3)

[1692] 10. The system of claim 1, wherein speech recognition is performed on the preprocessed data and the speech data is converted to text. [Explanation of symbols]

[1693] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A method to collect data such as emails, photos, videos, voice messages, social networking service logs, and letters from the electronic devices of deceased family members; a means for converting the collected data into a text format and preprocessing the data; A means of training a generative AI model based on the preprocessed data; and A means to build a chatbot using a trained generative AI model; A means of integrating the constructed chatbot into a communication interface; A system including:

2. 10. The system of claim 1, wherein optical character recognition is performed on the preprocessed data to convert the image and video data to text.

3. The system of claim 1 , wherein speech recognition is performed on the preprocessed data to convert the speech data to text.

Images