system

Abstract

We provide the system. [Solution] A means of providing an interface that allows users to input information about deceased persons, A means of sending user-entered information to the server, A means for the server to store the information it receives in a database, A means of generating a profile of a deceased person based on stored information, A means of generating a response using an AI model based on a message sent by a user, A system that includes means for sending the generated response to the user.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, while there is a desire to cherish memories and recollections of the deceased, there is a problem that there are few specific means to realize it. Conventional methods only involve saving photos, letters, and voice messages, and it is difficult to provide an interactive experience of "conversing" with the deceased. In addition, there are also concerns about privacy and security, and a system that can be used with confidence is required.

Means for Solving the Problems

[0005] This invention solves the above problem with a system that includes means for providing an interface for users to input information about deceased persons, means for transmitting the information entered by the user to a server, means for the server to store the received information in a database, means for generating a profile of the deceased person based on the stored information, means for generating a response using an AI model based on a message sent by the user, and means for sending the generated response to the user. Furthermore, by including means for providing a field for users to post episodes, means for transmitting episodes entered by users to a server, means for the server to store the received episodes in a database, and means for the AI ​​model to generate an adapted response based on previously stored episodes, users can enjoy conversations based on specific memories.In addition, by including means for the server to encrypt and store all data, means for executing the deletion of data requested by the user, and means for updating the user's privacy settings, a system that takes privacy and security into consideration can be provided.

[0006] A "user" refers to a person who uses this system, inputting information about deceased individuals and engaging in communication.

[0007] An "interface" refers to an input / output mechanism that allows a user to input information into a system or receive information from a system.

[0008] A "server" refers to a central computer system that receives, processes, and stores user-generated information in a database.

[0009] A "database" is a collection of information managed by a server, and refers to the place where user-entered profile information, episodes, and other data are stored.

[0010] A "profile" refers to a digital representation of a person created by aggregating information about a deceased individual.

[0011] An "AI model" refers to a set of algorithms that use artificial intelligence to generate natural responses based on user input.

[0012] "Response" refers to the reply or response that a system generates in response to user input.

[0013] An "episode" refers to information about memories or past events posted by users.

[0014] "Encryption" refers to the process of transforming information using special techniques to protect it, making it in a format that only authorized individuals can understand.

[0015] "Privacy settings" refer to options that allow users to set permissions and restrictions regarding the handling of their data and information. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8]It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the language used in the following description will be described.

[0019] In the following embodiments, a processor with a reference sign (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] To implement this invention, the system is configured as follows.

[0038] Creating a digital profile

[0039] Terminal operation:

[0040] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities.

[0041] Server processing:

[0042] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. This generates a digital profile of the deceased person.

[0043] Specific example:

[0044] Users enter a photo of the deceased "Taro Yamada," an audio message from him, and information about events he enjoyed. The server receives the submitted information and stores it in a database as Taro Yamada's digital profile.

[0045] AI reproduction

[0046] Terminal operation:

[0047] The user uses a text box on the device interface to enter a message to start a conversation. They enter the message and click the send button to begin the conversation.

[0048] Server processing:

[0049] The device sends a message, which is then received. The server then uses an AI model to analyze the received message and generate a natural response based on the user's digital profile. The generated response is sent to the device and displayed to the user.

[0050] Specific example:

[0051] The user types and sends the message "Hello, Taro." The server receives this message and, through its AI model, generates the response "Hello! How are you?" which is then sent to the device. The device then displays or plays this response aloud.

[0052] Memory Share

[0053] Terminal operation:

[0054] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the submit button.

[0055] Server processing:

[0056] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using an AI model based on the relevant information.

[0057] Specific example:

[0058] The user types, "Do you remember the day we went to the beach last summer?" The server searches the database for relevant episodes and generates a response saying, "Of course, the beach was really beautiful that day."

[0059] Privacy and security, ethical considerations

[0060] Data encryption:

[0061] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches.

[0062] Processing user requests:

[0063] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server then deletes the relevant data from its database and updates the settings.

[0064] Specific example:

[0065] The user requests to delete an episode. The device sends this request to the server, which deletes the episode from the database. A notification is sent to the device upon completion of the deletion, and the user is shown "Episode deleted."

[0066] In this way, the system provides users with an interactive experience with their deceased loved ones, creating opportunities to share memories. It also prioritizes privacy and security, ensuring a system that can be used with peace of mind.

[0067] The following describes the processing flow.

[0068] Creating a digital profile

[0069] Step 1:

[0070] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.).

[0071] Step 2:

[0072] The user completes the input and clicks the save button.

[0073] Step 3:

[0074] The terminal sends the entered information to the server in JSON format.

[0075] Step 4:

[0076] The server analyzes the received information and converts it to an appropriate format, such as Unicode.

[0077] Step 5:

[0078] The server saves the information to the database. After saving is complete, a notification is sent to the device confirming the save is finished.

[0079] Step 6:

[0080] The device receives a notification that saving is complete and displays "Digital profile saved" to the user.

[0081] AI reproduction

[0082] Step 1:

[0083] The user enters a message to start a conversation using the device's interface.

[0084] Step 2:

[0085] The user clicks the submit button.

[0086] Step 3:

[0087] The terminal sends the entered message to the server in JSON format.

[0088] Step 4:

[0089] The server receives the message and inputs it into the AI ​​model.

[0090] Step 5:

[0091] The server's AI model analyzes messages and generates appropriate responses. These responses include natural language processing and speech synthesis.

[0092] Step 6:

[0093] The server sends the generated response to the terminal in JSON format.

[0094] Step 7:

[0095] The device displays or plays the received response to the user.

[0096] Memory Share

[0097] Step 1:

[0098] Users enter text, images, and videos into fields on their device interface to post episodes.

[0099] Step 2:

[0100] The user clicks the submit button.

[0101] Step 3:

[0102] The device sends the posted data to the server in JSON format.

[0103] Step 4:

[0104] The server analyzes the received episodes, converts them to an appropriate format, and stores them in the database.

[0105] Step 5:

[0106] The user enters a question about a specific memory.

[0107] Step 6:

[0108] The server searches the database for relevant episodes and retrieves the corresponding data.

[0109] Step 7:

[0110] The server's AI model generates natural-sounding responses based on the acquired episodes.

[0111] Step 8:

[0112] The server sends the generated response to the terminal in JSON format.

[0113] Step 9:

[0114] The device displays or plays the received response to the user.

[0115] Privacy and security, ethical considerations

[0116] Step 1:

[0117] The server encrypts all incoming information and stores it in the database.

[0118] Step 2:

[0119] Users can request data deletion or changes to their privacy settings.

[0120] Step 3:

[0121] The device sends the user's request to the server in JSON format.

[0122] Step 4:

[0123] The server searches the database for the relevant data and deletes it or updates the privacy settings upon the user's request.

[0124] Step 5:

[0125] The server notifies the terminal in JSON format that the request processing is complete.

[0126] Step 6:

[0127] The device displays the received notification to the user.

[0128] (Example 1)

[0129] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0130] Currently, many users are seeking ways to digitally manage and share memories and information about deceased loved ones, but this is difficult with conventional systems. Furthermore, many systems lack sufficient consideration for privacy and security, making them unsafe to use. Additionally, there is a lack of technology to generate natural conversations based on user input.

[0131] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0132] In this invention, the server includes means for providing an interface to allow the user to input information about a deceased person; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a digital profile of the deceased person based on the stored information; means for generating a response using a generative AI model based on a message sent by the user; means for sending the generated response to the user; means for the server to store the received information using AES-256 encryption technology; means for deleting the relevant information from the database based on a deletion request from the user; and means for updating the user's privacy settings. As a result, the user can safely manage and share information about a deceased person and also enjoy natural conversations through the generative AI model.

[0133] A "user" refers to an individual who uses this system to input, manage, and share information about a deceased person.

[0134] "Interface" refers to the screens and operating methods that users use to access a system and input and manage information.

[0135] A "server" refers to a computer system that analyzes received information, stores it in a database, and generates responses using generative AI models.

[0136] A "database" refers to a digital information management system used to store and manage information about deceased individuals and episodes posted by users.

[0137] A "digital profile" refers to a collection of data that compiles information such as the name, age, photograph, audio sample, and personal anecdotes of a deceased person.

[0138] A "generative AI model" refers to an artificial intelligence model used to analyze user input messages and generate natural-sounding responses.

[0139] "AES-256 encryption technology" refers to a strong encryption technology used to keep received information secure.

[0140] A "deletion request" refers to the operation in which a user sends a request to a server to delete their own data.

[0141] "Privacy settings" refer to the options and actions that allow users to set the scope of visibility and access rights for their data.

[0142] To implement this invention, the system is configured as follows: First, the user inputs basic information about the deceased person using the terminal interface. This interface includes a text field, a file upload function, a voice recording function, and the like.

[0143] Creating a digital profile

[0144] Terminal operation:

[0145] The user opens the terminal interface and enters the deceased person's name, age, photo, audio sample, and personal anecdotes. Once the input is complete, they click the save button. At this point, the terminal converts the entered information into JSON format and sends it to the server as an HTTP request.

[0146] Server processing:

[0147] The server receives an HTTP request, parses the received content, and separates the data into individual fields. Photos are saved as binary data, and audio as audio files. The analyzed data is stored in a database using an RDBMS such as MySQL (registered trademark). This generates a digital profile of the deceased person.

[0148] Specific example:

[0149] The user enters information about "Deceased Person A" and uploads a photo and an audio message. When the user clicks the save button, the device sends this information to the server. The server analyzes the information and saves it to a database, completing the digital profile of "Deceased Person A".

[0150] AI reproduction

[0151] Terminal operation:

[0152] The user opens the terminal interface and types a message to start a conversation into the text box. After typing the message, they click the send button.

[0153] Server processing:

[0154] The terminal converts the message it sends into JSON format and sends it to the server. The server analyzes and processes the received message using a generation AI model (e.g., GPT-3® or ChatGPT®). The model generates a natural response, which the server converts back into JSON format and sends back to the terminal. The terminal displays the received response and plays it back as audio if necessary.

[0155] Specific example:

[0156] The user types and sends the message "Hello, Deceased Person A". The server receives this message and uses a generative AI model to generate the response "Hello! How are you?" and sends it to the device. The device displays and plays this response aloud.

[0157] Memory Share

[0158] Terminal operation:

[0159] Users use text fields and file upload functions to enter past memories and episodes. They enter the necessary information and click the submit button.

[0160] Server processing:

[0161] When the device recognizes a click of the post button, it converts the episode information into JSON format and sends it to the server as an HTTP request. The server stores the submitted episode in its database. If the user later initiates a conversation about a particular episode, the server searches and extracts the relevant episode from the database and generates a response using a generative AI model.

[0162] Specific example:

[0163] The user types, "Do you remember the day we went to the beach last summer?" The server searches the database for relevant episodes and generates a response saying, "Of course, the beach was really beautiful that day."

[0164] Privacy and security, ethical considerations

[0165] Data encryption:

[0166] The server stores all received information in a database using AES-256 encryption technology. This prevents unauthorized access and data breaches.

[0167] Processing user requests:

[0168] When a user requests data deletion or changes to privacy settings, the device converts this data into JSON format and sends it to the server as an HTTP request. The server receives the request, deletes the data from the database, and updates the settings. If the deletion or update is successful, the server sends a notification back to the device, and the user is shown a message confirming the completion of the deletion.

[0169] Specific example:

[0170] The user requests to delete an episode. The device sends this request to the server, which then deletes the data from the database. A notification is sent to the device upon completion of the deletion, displaying the message "Episode deleted."

[0171] Example of a prompt

[0172] Creating a digital profile: The deceased person's name, age, photo, voice sample, personal anecdotes, etc., are entered on a device and saved to the server's database.

[0173] AI-powered reproduction: The user enters a message, and the server generates a natural response using an AI model.

[0174] Memory sharing: Users post past episodes from their devices, and the server saves them to a database.

[0175] Privacy and Security: The server encrypts all incoming information and processes user data deletion requests.

[0176] In this way, the system provides users with an interactive experience with deceased loved ones, creating opportunities to share memories, while also ensuring a safe and secure system that respects privacy and security.

[0177] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0178] Creating a digital profile

[0179] Step 1:

[0180] The user opens the device's interface and enters the deceased person's name, age, photo, voice sample, personal anecdotes, etc. The entered information is temporarily stored in the device's local memory.

[0181] Input: User enters name, age, photo, voice sample, and episode.

[0182] Output: Information temporarily stored in the terminal's local memory.

[0183] Step 2:

[0184] After the user has entered all the information, they click the save button. The device converts the entered information into JSON format and sends it to the server as an HTTP request.

[0185] Input: Click the save button.

[0186] Output: Information in JSON format sent to the server.

[0187] Step 3:

[0188] The server receives an HTTP request, parses the received content, and divides the data into individual fields. For example, it might save the name as a string field, the photo as binary data, and the audio as an audio file.

[0189] Input: Information sent to the server in JSON format.

[0190] Output: Data for each field analyzed within the server.

[0191] Step 4:

[0192] The server saves the analyzed data to a database using an RDBMS such as MySQL. If the save is successful, the server sends a success message to the terminal in JSON format.

[0193] Input: Data separated into each field.

[0194] Output: Data stored in the database and a success message sent to the terminal.

[0195] AI reproduction

[0196] Step 1:

[0197] The user opens the terminal interface and types a message to start a conversation into the text box. After typing the message, they click the send button.

[0198] Input: User input of a text message.

[0199] Output: Messages temporarily stored within the device.

[0200] Step 2:

[0201] The device recognizes the click of the send button, converts the message to JSON format, and sends it to the server as an HTTP request.

[0202] Input: Click the submit button.

[0203] Output: A JSON-formatted message sent to the server.

[0204] Step 3:

[0205] The server analyzes and processes received messages using a generating AI model (e.g., GPT-3 or ChatGPT). The model then generates a natural-sounding response.

[0206] Input: A JSON-formatted message sent to the server.

[0207] Output: Response generated through the generative AI model.

[0208] Step 4:

[0209] The server converts the generated response into JSON format and sends it to the terminal as an HTTP response. The terminal displays the received response and plays it aloud if necessary.

[0210] Input: The generated response.

[0211] Output: A JSON-formatted response sent to the terminal, along with its display and audio playback on the terminal.

[0212] Memory Share

[0213] Step 1:

[0214] The user opens the terminal interface and uses text fields and file upload functions to enter past memories and episodes. They enter the necessary information and click the submit button.

[0215] Input: User input of episodes and files.

[0216] Output: Episode information temporarily stored on the device.

[0217] Step 2:

[0218] The device recognizes the click of the post button, converts the episode information into JSON format, and sends it to the server as an HTTP request.

[0219] Input: Click the submit button.

[0220] Output: Episode information in JSON format sent to the server.

[0221] Step 3:

[0222] The server saves the received episode to the database. After saving, the server sends a success message to the terminal in JSON format.

[0223] Input: Episode information in JSON format sent to the server.

[0224] Output: Episode information stored in the database and a success message sent to the terminal.

[0225] Step 4:

[0226] If a user later initiates a conversation based on a specific episode, the server searches and extracts the relevant episode from the database and generates a response using a generative AI model.

[0227] Input: User-submitted messages based on their experiences.

[0228] Output: Natural responses generated through a generative AI model.

[0229] Privacy and security, ethical considerations

[0230] Step 1:

[0231] The server stores all received information in a database using AES-256 encryption technology. This prevents unauthorized access and data breaches.

[0232] Input: Information sent to the server.

[0233] Output: Database storage encrypted using AES-256 technology.

[0234] Step 2:

[0235] When a user requests data deletion or changes to privacy settings, the device converts this information into JSON format and sends it to the server as an HTTP request.

[0236] Input: User request for deletion or change of privacy settings.

[0237] Output: A JSON-formatted request sent to the server.

[0238] Step 3:

[0239] The server receives the request, deletes the relevant data from the database, and updates the settings. If the deletion or update is successful, the server sends a success message to the terminal.

[0240] Input: Deletion or privacy setting change request in JSON format.

[0241] Output: Data deleted from the database, or updated privacy settings, and a success message notified to the user.

[0242] (Application Example 1)

[0243] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0244] Currently, many users wish to share memories of deceased family and friends and maintain emotional connections through dialogue. However, existing systems lack the technology to meticulously profile information about deceased individuals and generate natural conversations using AI. Furthermore, concerns about privacy and security necessitate a safe and secure system. This invention aims to solve these problems and provide a system that allows users to engage in natural virtual dialogues with deceased individuals.

[0245] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0246] In this invention, the server includes means for providing an interface to allow the user to input information about a deceased person; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a profile of the deceased person based on the stored information; means for generating a response using a generative AI model based on a message sent by the user; means for sending the generated response to the user; means for encrypting and decrypting the profile data; means for storing past episodes and having the generative AI model generate responses based on these; and means for generating and sending prompt messages to the user's terminal. This enables the user to have a natural conversation with the deceased person and share memories. Furthermore, it enables safe use that takes privacy and security into consideration.

[0247] A "user" is someone who uses the system to input information about a deceased person and then engages in subsequent conversations and shares stories about them.

[0248] An "interface" refers to the screen or control panel that a user uses to operate a device, input information, and transmit it.

[0249] A "profile" is a collection of information that includes the name, age, photograph, audio sample, and personal anecdotes of a deceased person.

[0250] A "generative AI model" is an artificial intelligence model used to generate natural-sounding responses based on user input messages and referencing the profiles of deceased individuals.

[0251] A "response" refers to the reply or reaction from a generative AI model to a message sent by a user.

[0252] A "database" is an electronic information management system used by a server to store information and episodes it receives.

[0253] "Encryption" is the process of transforming data using special techniques to make it unreadable, in order to protect it from unauthorized access.

[0254] "Decryption" is the process of restoring encrypted data to its original, readable state.

[0255] An "episode" is a user-submitted memory or event related to a deceased person.

[0256] A "prompt sentence" is an input sentence for a generative AI model, and it is a sentence necessary to generate an accurate response based on the user's message.

[0257] A "terminal" refers to a user-operated device such as a smartphone, and is used to provide the interface for this system.

[0258] The system necessary to implement this invention is configured as follows: The system mainly consists of a user terminal, a server, a database, and a generative AI model.

[0259] Creating a digital profile

[0260] The user first uses the terminal's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). Specifically, they use the text fields, file upload function, and voice recording function on the terminal's screen. This information is sent to the server when the user clicks the save button. The server analyzes the received information and stores it in a database, generating a digital profile of the deceased person.

[0261] AI reproduction

[0262] The user enters a message to initiate a conversation through the terminal's interface. The server receives the message sent from the terminal and analyzes it using a generative AI model. It then generates a natural response based on the user's digital profile and sends it to the terminal for display to the user.

[0263] Memory Share

[0264] Users use text fields and file upload functions to post past memories and episodes. By entering the necessary information and clicking the submit button, the information is sent to the server. The server stores the received episodes in a database. Subsequently, when a user starts a conversation based on a particular episode, the server searches for previously stored episodes and generates a response using a generative AI model based on the relevant information.

[0265] Privacy and security, ethical considerations

[0266] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data leaks. Furthermore, if a user requests data deletion or changes to privacy settings, the device sends this request to the server, which then deletes the relevant data from the database and updates the settings.

[0267] To give a specific example, when a user types and sends the message "Hello, Taro," the server receives this message, generates a response "Hello! How are you?" through a generative AI model, and sends and displays it on the device.

[0268] Furthermore, if you want to talk about past memories, you can type something like, "Do you remember the day we went to the beach last summer?" The server will search its database for relevant episodes and generate a response like, "Of course, the beach was really beautiful that day."

[0269] Specific examples of hardware and software include smartphones as terminals (iOS, ANDROID®), Flask as a server-side framework, SQLite and PostgreSQL as databases, and GPT-3 (using the transformers library from Hugging Face) as a generative AI model.

[0270] Specific examples of prompt messages include user messages such as "What are you doing today?". Questions like "How are you doing?" and messages that evoke past events, such as "Do you remember the day we went to the beach last summer?" are also used as prompt messages.

[0271] This allows users to have natural conversations with deceased loved ones and share memories. Furthermore, it enables safe use with consideration for privacy and security.

[0272] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0273] Step 1:

[0274] The user operates the device's interface to enter basic information about the deceased person. This interface includes text fields, a file upload function, and a voice recording function. Once all the information is entered, the user clicks the save button to confirm the input. Examples of input data include name, age, photo, voice sample, and personal anecdotes.

[0275] Step 2:

[0276] The device sends information entered by the user to the server. During this process, the device converts the input data into JSON format and transmits it securely using the HTTPS protocol. The data received by the server includes information such as name, age, photo, voice sample, and episode.

[0277] Step 3:

[0278] The server analyzes the received information and stores it in the database. The server encrypts the data using an encryption method and stores it in an appropriate format in each field to maintain data integrity later. This data is stored in a SQL database (e.g., PostgreSQL). The input here is user data, and the output is the encrypted data stored in the database.

[0279] Step 4:

[0280] The user inputs a message to start a conversation at the interface of the terminal. By entering the message in the text box and clicking the send button, the message is sent to the server. Examples of specific prompt texts include "Hello, how are you?"

[0281] Step 5:

[0282] The server analyzes the message received from the terminal. First, it decrypts and retrieves the target profile from the database and formats it into an appropriate format for input to the generative AI model (e.g., GPT-3). The input data is the message from the user and the profile information, and the output is the prompt text provided to the generative AI model.

[0283] Step 6:

[0284] The server uses the generative AI model to generate a natural response from the prompt text. The generative AI model generates an appropriate response based on the profile information and the user's message. Through this process, a response such as "Hello! Are you doing well?" is obtained.

[0285] Step 7:<00009​The server sends the generated response to the user's terminal. At this time, the response is encoded in JSON format and securely sent using the HTTPS protocol. The input data is the generated response message, and the output data is the display of the response on the user's terminal.

[0287] Step 8:

[0288] The terminal displays the received response to the user. A text - formatted message is displayed on the UI or played as audio. This allows the user to continue the conversation.

[0289] Step 9:

[0290] The user uses a text field or file - upload function to post past memories or episodes. The input data is confirmed with the post button, and the terminal sends the data to the server. The data sent includes memories or episodes.

[0291] Step 10:

[0292] The server saves the received episodes in the database. Also, the data is encrypted for storage at this time. The input data is episode information, and the output data is the episode information encrypted and saved in the database.

[0293] Step 11:

[0294] When the user starts a conversation based on past episodes, the server searches the database for relevant episodes and uses a generative AI model to generate a corresponding response. For example, if the user enters "Do you remember the day we went to the beach last summer?", the server searches for relevant episodes and generates a response like "Of course, the sea that day was really beautiful."

[0295] Step 12:

[0296] The server processes data encryption, decryption, and deletion requests based on the user's privacy and security settings. This prevents unauthorized access and data breaches, ensuring secure use.

[0297] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0298] To carry out this invention, the system consists of the following elements.

[0299] Creating a digital profile

[0300] Terminal operation:

[0301] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities.

[0302] Server processing:

[0303] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. This generates a digital profile of the deceased person.

[0304] Specific example:

[0305] Users input a photo of the deceased "Ichiro Tanaka," along with audio messages from his lifetime and information about events he enjoyed. The server receives the submitted information and stores it in a database as Ichiro Tanaka's digital profile.

[0306] AI reproduction

[0307] Terminal operation:

[0308] The user inputs a message to start a conversation through the terminal interface. By clicking the send button, the user starts the conversation.

[0309] Server processing:

[0310] Receive the message sent by the terminal. Next, the server analyzes the received message using the AI model and generates a natural response based on the digital profile. The generated response is sent to the terminal and displayed to the user.

[0311] Specific example:

[0312] The user inputs and sends a message "Hello, Ichiro". The server receives this message, generates a response "Hello! Are you doing well?" through the AI model, and sends it to the terminal. The terminal displays it or plays it as audio.

[0313] Memory sharing

[0314] Terminal operation:

[0315] The user uses a text field or file upload function to post past memories or episodes. Enter the necessary information and click the post button.

[0316] Server processing:

[0317] Receive the posted episodes or memories and save them in the database. Then, when the user starts a conversation based on a specific episode, the server searches for the previously saved episode and generates a response using the AI model based on the relevant information.

[0318] Specific example:

[0319] The user types, "Do you remember last year's fireworks display?" The server searches the database for relevant episodes and generates a response saying, "Of course, the fireworks that day were really beautiful."

[0320] Emotion recognition by an emotion engine

[0321] Terminal operation:

[0322] When a user enters a text message, an engine works to analyze the emotions behind that message.

[0323] Server processing:

[0324] When receiving a message sent from a device, the emotion engine analyzes the message content and recognizes the emotional state (e.g., joy, sadness, anger). Based on the recognized emotion, the server adjusts the AI ​​model, changing the tone and content of the response.

[0325] Specific example:

[0326] The user sends the message, "I'm really sad today." The server's emotion engine analyzes this message and recognizes that the user is in a state of sadness. Based on this information, the server generates a comforting response, "It's okay, I'll listen," and sends it to the device.

[0327] Privacy and Security

[0328] Data encryption:

[0329] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches.

[0330] Processing user requests:

[0331] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server then deletes the relevant data from the database and updates the settings.

[0332] Specific example:

[0333] The user requests to delete an episode. The device sends this request to the server, which deletes the episode from the database. A notification is sent to the device upon completion of the deletion, and the user is shown "Episode deleted."

[0334] In this way, the system provides users with an interactive experience with their deceased loved ones, creating opportunities to share memories. Furthermore, by utilizing an emotion engine, it can respond in accordance with the user's emotions, resulting in a more personalized experience. With consideration for privacy and security, it provides a system that can be used with peace of mind.

[0335] The following describes the processing flow.

[0336] Creating a digital profile

[0337] Step 1:

[0338] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.).

[0339] Step 2:

[0340] The user completes the input and clicks the save button.

[0341] Step 3:

[0342] The terminal sends the entered information to the server in JSON format.

[0343] Step 4:

[0344] The server analyzes the received information and converts it to an appropriate format, such as Unicode.

[0345] Step 5:

[0346] The server saves the information to the database. After saving is complete, a notification is sent to the device confirming the save is finished.

[0347] Step 6:

[0348] The device receives a notification that saving is complete and displays "Digital profile saved" to the user.

[0349] AI reproduction

[0350] Step 1:

[0351] The user enters a message to start a conversation using the device's interface.

[0352] Step 2:

[0353] The user clicks the submit button.

[0354] Step 3:

[0355] The terminal sends the entered message to the server in JSON format.

[0356] Step 4:

[0357] The server receives the message and inputs it into the AI ​​model.

[0358] Step 5:

[0359] The server's AI model analyzes messages and generates appropriate responses. These responses include natural language processing and speech synthesis.

[0360] Step 6:

[0361] The server sends the generated response to the terminal in JSON format.

[0362] Step 7:

[0363] The device displays or plays the received response to the user via audio.

[0364] Memory Share

[0365] Step 1:

[0366] Users enter text, images, and videos into fields on their device interface to post episodes.

[0367] Step 2:

[0368] The user clicks the submit button.

[0369] Step 3:

[0370] The device sends the posted data to the server in JSON format.

[0371] Step 4:

[0372] The server analyzes the received episodes, converts them to an appropriate format, and stores them in the database.

[0373] Step 5:

[0374] The user enters a question about a specific memory.

[0375] Step 6:

[0376] The server searches the database for relevant episodes and retrieves the corresponding data.

[0377] Step 7:

[0378] The server's AI model generates natural-sounding responses based on the acquired episodes.

[0379] Step 8:

[0380] The server sends the generated response to the terminal in JSON format.

[0381] Step 9:

[0382] The device displays or plays aloud the response it receives to the user.

[0383] Emotion recognition by an emotion engine

[0384] Step 1:

[0385] When a user enters a text message, an engine works to analyze the emotions behind that message.

[0386] Step 2:

[0387] The terminal sends the entered message to the server in JSON format.

[0388] Step 3:

[0389] The server receives the message, and the emotion engine analyzes the message's content to recognize the emotional state (e.g., joy, sadness, anger).

[0390] Step 4:

[0391] Based on the emotions the server recognizes, it adjusts the AI ​​model, changing the tone and content of its responses.

[0392] Step 5:

[0393] The server sends the generated response to the terminal in JSON format.

[0394] Step 6:

[0395] The device displays or plays the received response to the user via audio.

[0396] Privacy and Security

[0397] Step 1:

[0398] The server encrypts all incoming information and stores it in the database.

[0399] Step 2:

[0400] Users can request data deletion or changes to their privacy settings.

[0401] Step 3:

[0402] The device sends the user's request to the server in JSON format.

[0403] Step 4:

[0404] The server searches the database for the relevant data and deletes it or updates the privacy settings upon the user's request.

[0405] Step 5:

[0406] The server notifies the terminal in JSON format that the request processing is complete.

[0407] Step 6:

[0408] Displays notifications received by the device to the user.

[0409] (Example 2)

[0410] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0411] In modern society, there is a need for systems that effectively preserve memories of the deceased and allow users to communicate with them interactively. Natural conversations based on the deceased's emotions and past events are particularly important, while ensuring data privacy and security remains a crucial challenge. Furthermore, technologies that can quickly and accurately analyze user input and generate appropriate responses are essential.

[0412] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0413] In this invention, the server includes means for analyzing messages sent by the user and generating responses using a generative AI model, means for analyzing the emotional state of the user's input message and adjusting the response based on the emotional state, and means for encrypting all received information and storing it in a database. This enables users to communicate with deceased loved ones in an interactive manner, share personal memories and episodes, and use the system with peace of mind.

[0414] A "user" is an individual who uses the system to input information about the deceased or to engage in conversations with them.

[0415] A "deceased person" is someone whose profile and memories are saved in the system, and who has passed away and who can be the subject of virtual conversations with the user.

[0416] An "interface" refers to the user interface or tools that a user uses to input information about a deceased person.

[0417] A "server" is a central computer system that receives information sent by users, processes it, and stores it in a database.

[0418] A "database" is a computer system used to structurally store and manage data such as information and anecdotes about a deceased person.

[0419] A "digital profile" is a collection of data, including the deceased person's name, age, photograph, audio sample, and personal anecdotes, which is stored in a database.

[0420] A "generative AI model" is an artificial intelligence algorithm used to analyze user messages and generate natural-sounding responses.

[0421] "Emotional state" refers to the type and intensity of emotions analyzed from the user's input messages.

[0422] "Encryption" is a technology that transforms data using specific algorithms to protect it from unauthorized access and leakage.

[0423] An "episode" is a detailed description or text of memories or events related to the deceased that the user has entered.

[0424] A "response" is a response message generated by a generative AI model and sent to the user.

[0425] "Privacy settings" are settings that allow users to define the scope of their data's visibility and use within the system.

[0426] This invention relates to a system that allows users to input information about a deceased person and engage in interactive conversations using a generated AI model based on that information. This system mainly consists of a terminal, a server, and a database.

[0427] Creating a digital profile

[0428] Terminal operation:

[0429] The user uses the device's interface to enter basic information about the deceased (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities. For example, the user might enter the deceased's name, "Mr. / Ms. A," a photo, a voice message, and their favorite events.

[0430] Server processing:

[0431] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. For example, the server receives a photo and voice message from "Person A" and stores them in the database to generate Person A's digital profile.

[0432] AI reproduction

[0433] Terminal operation:

[0434] The user enters a message to start a conversation using the terminal's interface. For example, they might type "Hello, A" and click the send button.

[0435] Server processing:

[0436] The server receives a message sent by the terminal. Next, the server analyzes the received message using a generative AI model (e.g., GPT-4®) and generates a natural response based on the digital profile. The generated response is sent to the terminal and displayed to the user. For example, if the server receives the message "Hello, A," the generative AI model generates the response "Hello! How are you?" and sends it to the terminal.

[0437] Memory Share

[0438] Terminal operation:

[0439] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the post button. For example, a user might enter text and photos on the theme of "Last year's fireworks display."

[0440] Server processing:

[0441] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using a generative AI model based on the relevant information. For example, upon receiving the message, "Do you remember last year's fireworks display?", the server generates the response, "Of course, the fireworks that day were really beautiful."

[0442] Emotion recognition by an emotion engine

[0443] Terminal operation:

[0444] When a user types a text message, an engine works to analyze the emotion behind that message. For example, the user might type, "I'm really sad today."

[0445] Server processing:

[0446] When receiving a message sent from a device, the emotion engine analyzes the message content and recognizes the emotional state (e.g., joy, sadness, anger). Based on the recognized emotion, the server adjusts the generative AI model to change the tone and content of the response. For example, in response to a message that says, "I'm really sad today," the server might generate a comforting response such as, "It's okay, I'll listen."

[0447] Privacy and Security

[0448] Data encryption:

[0449] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches. For example, all transmitted data is encrypted using the TLS / SSL protocol.

[0450] Processing user requests:

[0451] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server deletes the relevant data from the database and updates the settings. For example, if a user requests to delete a specific episode, the server deletes that episode from the database and sends a message to the device stating, "Episode deleted."

[0452] This system allows users to enjoy interactive experiences with deceased loved ones and share personal memories. Furthermore, by utilizing an emotion engine and generative AI models, it provides personalized responses tailored to the user's emotions. In addition, the system is designed with privacy and security in mind, ensuring safe use.

[0453] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0454] Creating a digital profile

[0455] Terminal operation:

[0456] Step 1:

[0457] The user opens the device's interface. The user enters the deceased's name and age in the text fields on the interface, adds a photo using the file upload function, and records a voice sample using the voice recording function. This becomes the input data.

[0458] Server processing:

[0459] Step 2:

[0460] Once the user completes the information entry and clicks the save button, the device sends the entered information (text, photos, audio) to the server in JSON format. This is the input process.

[0461] Step 3:

[0462] The server parses the received JSON data and extracts each field (name, age, photo, audio, etc.). This parsing process prepares the data structure for storage in the database. Next, the server saves the parsed data to the database in the appropriate format. Specifically, it saves the name and age as text data, and the photo and audio as binary data. This is the output.

[0463] AI reproduction

[0464] Terminal operation:

[0465] Step 1:

[0466] The user opens a chat on their device's interface and enters a message such as "Hello, A" into the text field. This is the input data.

[0467] Step 2:

[0468] When the user clicks the send button, the terminal sends the entered message to the server in JSON format. This is the input.

[0469] Server processing:

[0470] Step 3:

[0471] The server analyzes the received message and compares it to a digital profile. Based on the comparison results, the server uses a generative AI model (e.g., GPT-4) to create a prompt for generating an appropriate response. This prompt might take the form of, for example, "The user said 'Hello, Mr. / Ms. A' to the deceased. Please respond as the deceased." This is data processing.

[0472] Step 4:

[0473] The generated prompt is used to run the generative AI model, which then produces a natural-sounding response (e.g., "Hello! How are you?"). This is the output.

[0474] Step 5:

[0475] The server sends the generated response to the terminal in JSON format. This is also output.

[0476] Terminal operation:

[0477] Step 6:

[0478] The terminal displays the received response in the chat interface for the user to see. It also plays it back as audio if necessary. This is the final output.

[0479] Memory Share

[0480] Terminal operation:

[0481] Step 1:

[0482] Users input past episodes and memories using text fields or file upload functions. For example, they might input an episode titled "Last year's fireworks display" along with a photo. This is the input data.

[0483] Step 2:

[0484] When a user clicks the "Submit" button, the device sends the entered episode data to the server in JSON format. This is the input.

[0485] Server processing:

[0486] Step 3:

[0487] The server analyzes the received episode data and stores it in the database. The analysis process extracts the episode text and photo data, which are then inserted into the appropriate records in the database. This is the output.

[0488] Step 4:

[0489] When a user sends the message "Do you remember last year's fireworks display?" to the server, the server searches its database for relevant episodes. Specifically, it searches using the keyword "fireworks display" and extracts the corresponding data. This is the input data search.

[0490] Step 5:

[0491] Based on the episode information extracted by the server, a generative AI model is used to generate a response such as, "Of course, the fireworks that day were really beautiful." This is data processing.

[0492] Step 6:

[0493] The server sends the generated response to the terminal, which then displays it to the user. This is the final output.

[0494] Emotion recognition by an emotion engine

[0495] Terminal operation:

[0496] Step 1:

[0497] The user types a text message and sends it saying, "I'm really sad today." This is the input data.

[0498] Step 2:

[0499] When the user clicks the send button, the device sends the message to the server in JSON format. This is the input.

[0500] Server processing:

[0501] Step 3:

[0502] The server analyzes the received message and uses an emotion engine to recognize the user's emotional state as "sad." This emotion analysis process is a data calculation.

[0503] Step 4:

[0504] After the emotional state is recognized, the server adjusts the generating AI model to produce a comforting response such as, "It's okay, I'll listen." This is the output.

[0505] Step 5:

[0506] The server sends the generated response to the terminal, which then displays it to the user. It also plays it back as audio if necessary. This is the final output.

[0507] Privacy and Security

[0508] Data encryption:

[0509] Step 1:

[0510] The server encrypts all incoming information. Specifically, it encrypts the information using the SSL / TLS protocol and stores it in the database. This is the process of protecting the input data.

[0511] Processing user requests:

[0512] Step 2:

[0513] The user requests data deletion or changes to privacy settings from their device. For example, they might request, "I want to delete a specific episode." This is the input data.

[0514] Step 3:

[0515] The terminal sends a request to the server. This is the input.

[0516] Server processing:

[0517] Step 4:

[0518] The server analyzes the received request and deletes the corresponding data from the database. Specifically, it searches for the relevant episode based on the episode ID included in the request and physically deletes it from the database. This is data processing and output.

[0519] Step 5:

[0520] After the deletion is complete, the server notifies the terminal of the result. A message "Episode deleted" is generated and sent to the terminal. This is the final output.

[0521] This system provides users with an interactive experience with their deceased loved ones, creating opportunities to share memories. By utilizing an emotion engine and generative AI models, it can respond to the user's emotions, resulting in a more personalized experience. Furthermore, privacy and security measures are in place, ensuring safe and secure use.

[0522] (Application Example 2)

[0523] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0524] Traditional interactive systems for sharing memories of deceased loved ones lacked the ability to generate responses that took into account the user's emotions and location. This made it difficult to provide information about the deceased and their past in a more natural and personalized way. Furthermore, ensuring the privacy and security of information was insufficient. As a result, users were unable to enjoy a high level of immersion and security.

[0525] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for providing an interface to allow the user to input information about the deceased; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a digital profile of the deceased based on the stored information; means for generating a response using a generation AI model based on a message sent by the user; means for acquiring information at a specific spot using the customer's location information and generating a response; and means for transmitting the generated response to the user and playing it back by display or audio. This makes it possible to provide natural interaction tailored to the user's individual circumstances and location, ensuring the privacy and security of information while allowing for a higher level of immersion.

[0526] 1. A "user" is a person who uses this system to input, transmit, and manage information about the deceased.

[0527] 2. "Interface" refers to the screens, fields, and means of operating them that are necessary for a user to input information about the deceased.

[0528] 3. A "server" is a central computer system that processes information received from users and stores it in a database.

[0529] 4. A "database" is a data storage system used by a server to store, search, and manage information it receives.

[0530] 5. A "digital profile" refers to virtual personal data that includes basic information and memories of the deceased.

[0531] 6. A "generative AI model" is an artificial intelligence algorithm that analyzes messages sent by users and generates appropriate responses.

[0532] 7. A "response" is the answer or response that a server generates in response to user input or actions.

[0533] 8. "Location information" refers to GPS data that indicates the user's current location.

[0534] 9. A "spot" is a location associated with specific information or a particular response.

[0535] 10. "Audio playback" refers to a method of delivering a message generated as a response to the user as audio.

[0536] 11. An "episode" is a specific story or memory shared by a user about a deceased person or past events.

[0537] 12. A "message" is a piece of text or a request that a user inputs through the interface and sends to the server.

[0538] 13. "Sentiment analysis" refers to the process of recognizing a user's emotional state from their messages and statements.

[0539] 14. "Personalization" means providing customized support tailored to each individual user.

[0540] 15. "Privacy settings" are settings that allow users to decide how their information is shared and how their data is handled.

[0541] 16. "Encryption" is a technique that converts data into a format that cannot be read in order to securely protect it.

[0542] 17. The "episode submission field" is an input field where users can enter past events or memories.

[0543] The system for carrying out this invention consists of the following hardware and software.

[0544] Hardware and software used

[0545] Hardware: Smartphones, smart glasses, head-mounted displays

[0546] Software: AWS® (Amazon Web Services), MySQL, Python, TENSORFLOW®, OpenCV

[0547] System Configuration

[0548] The system primarily consists of user terminals, servers, and databases. The operation and role of each major component are described in detail below.

[0549] User terminal

[0550] The user terminal uses devices such as smartphones, smart glasses, and head-mounted displays. The user terminal provides an interface for inputting information about the deceased and functions for transmitting that information. This allows users to easily input data and send it to the server.

[0551] server

[0552] The server processes information received from the user and generates a digital profile. It also uses a generative AI model to produce natural-sounding responses based on the user's input messages. The server stores the received information in a MySQL database and searches this data as needed to generate the best possible answers.

[0553] database

[0554] The database plays a role in securely storing information received from the server. The database contains encrypted data such as information about deceased individuals, user-submitted episodes, and past message histories.

[0555] Processing flow

[0556] The user first uses an interface to enter information about the deceased. This interface includes text fields, a file upload function, and a voice recording function. The information entered by the user is sent to the server, analyzed, and stored in a database. This generates a digital profile of the deceased.

[0557] When a user stands at a specific location, the server uses GPS information from their smartphone or smart glasses to retrieve specific information. Based on this, a generative AI model generates a relevant response and sends it to the user's device. The user's device then displays or plays this response as text or audio.

[0558] Emotion analysis and personalization

[0559] When a user enters a message, the sentiment analysis engine activates to recognize the user's emotions. Based on these emotions, the generative AI model generates a personalized response. For example, if a user enters "I'm really sad today," the server performs sentiment analysis and, depending on the user's emotional state, generates a comforting response such as "It's okay, I'll listen" and sends it to the user's device.

[0560] Privacy and Security

[0561] All data is end-to-end encrypted and stored in the database. If a user requests data deletion or changes to privacy settings, the server responds immediately, deleting the relevant data from the database or updating the settings. This ensures users can use the system with peace of mind.

[0562] Examples of specific cases and prompt statements

[0563] As a concrete example, if a profile of a former regular customer named "Ichiro Tanaka" is registered, and a customer at the store says using smart glasses, "I've found Ichiro's usual seat," the AI ​​will respond, "Really? Ichiro used to drink coffee here often."

[0564] Example of a prompt:

[0565] "Tell me about this place."

[0566] "Tell me about Christmas events from the past."

[0567] "Please share your memories from here."

[0568] In this way, the system provides users with a personalized, interactive experience, deepening their connection with deceased loved ones and past events. Furthermore, security measures ensure that users can use the system with peace of mind.

[0569] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0570] Step 1:

[0571] The user uses the terminal interface to enter basic information about the deceased (name, age, photo, audio sample, personal anecdotes, etc.). The entered information is sent from the user's terminal to the server. Specifically, the user enters information into a text field and clicks the "Send" button. The input is text or multimedia files (photo, audio), and the output is a data packet sent to the server.

[0572] Step 2:

[0573] The server parses the received information and stores it in a MySQL database. The server converts the input data into the appropriate format and inserts it into the database using SQL statements. The input is the user's data sent, and the output is a digital profile stored in the database.

[0574] Step 3:

[0575] When a user stands at a specific location, the server uses the device's GPS information to obtain their location. The device sends current coordinate data to the server, which then uses this data to search for relevant information. The input is the GPS data sent from the device, and the output is the relevant information returned by the server.

[0576] Step 4:

[0577] The server generates a response using a generative AI model based on relevant information. Specifically, the server searches for relevant episodes in the database and inputs them as prompts into the generative AI model. The server then analyzes the text generated by the model and produces an appropriate response. The input is relevant information based on location data, and the output is the generated response.

[0578] Step 5:

[0579] The server sends the generated response to the user's terminal, which then displays or plays it as text or audio. Specifically, the server sends the generated response to the terminal, and the terminal presents the received data to the user in an appropriate format. The input is the generated response, and the output is the text display or audio playback to the user.

[0580] Step 6:

[0581] When a user enters and sends an additional message, the server receives it and uses its sentiment analysis engine to recognize the emotion. Based on the analysis results, the server adjusts its AI model to generate a response appropriate to the emotion. The input is the user's message, and the output is the emotion-appropriate response.

[0582] Step 7:

[0583] The server encrypts all data and securely stores it in the database. The input is the data to be stored, and the output is the encrypted data. When a user requests data deletion or changes to privacy settings, the server reflects this in the database and updates the settings. Specifically, it parses the user's request and deletes or updates the corresponding data from the database. The input is the user's request, and the output is the updated database state.

[0584] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0585] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0586] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0587] [Second Embodiment]

[0588] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0589] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0590] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0591] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0592] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0593] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0594] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0595] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0596] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0597] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0598] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0599] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0600] To implement this invention, the system is configured as follows.

[0601] Creating a digital profile

[0602] Terminal operation:

[0603] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities.

[0604] Server processing:

[0605] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. This generates a digital profile of the deceased person.

[0606] Specific example:

[0607] Users enter a photo of the deceased "Taro Yamada," an audio message from him, and information about events he enjoyed. The server receives the submitted information and stores it in a database as Taro Yamada's digital profile.

[0608] AI reproduction

[0609] Terminal operation:

[0610] The user uses a text box on the device interface to enter a message to start a conversation. They enter the message and click the send button to begin the conversation.

[0611] Server processing:

[0612] The device sends a message, which is then received. The server then uses an AI model to analyze the received message and generate a natural response based on the user's digital profile. The generated response is sent to the device and displayed to the user.

[0613] Specific example:

[0614] The user types and sends the message "Hello, Taro." The server receives this message and, through its AI model, generates the response "Hello! How are you?" which is then sent to the device. The device then displays or plays this response aloud.

[0615] Memory Share

[0616] Terminal operation:

[0617] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the submit button.

[0618] Server processing:

[0619] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using an AI model based on the relevant information.

[0620] Specific example:

[0621] The user types, "Do you remember the day we went to the beach last summer?" The server searches the database for relevant episodes and generates a response saying, "Of course, the beach was really beautiful that day."

[0622] Privacy and security, ethical considerations

[0623] Data encryption:

[0624] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches.

[0625] Processing user requests:

[0626] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server then deletes the relevant data from its database and updates the settings.

[0627] Specific example:

[0628] The user requests to delete an episode. The device sends this request to the server, which deletes the episode from the database. A notification is sent to the device upon completion of the deletion, and the user is shown "Episode deleted."

[0629] In this way, the system provides users with an interactive experience with deceased loved ones and creates opportunities to share memories. It also prioritizes privacy and security, ensuring a system that can be used with peace of mind.

[0630] The following describes the processing flow.

[0631] Creating a digital profile

[0632] Step 1:

[0633] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.).

[0634] Step 2:

[0635] The user completes the input and clicks the save button.

[0636] Step 3:

[0637] The terminal sends the entered information to the server in JSON format.

[0638] Step 4:

[0639] The server analyzes the received information and converts it to an appropriate format, such as Unicode.

[0640] Step 5:

[0641] The server saves the information to the database. After saving is complete, a notification is sent to the device confirming the save is finished.

[0642] Step 6:

[0643] The device receives a notification that saving is complete and displays "Digital profile saved" to the user.

[0644] AI reproduction

[0645] Step 1:

[0646] The user enters a message to start a conversation using the device's interface.

[0647] Step 2:

[0648] The user clicks the submit button.

[0649] Step 3:

[0650] The terminal sends the entered message to the server in JSON format.

[0651] Step 4:

[0652] The server receives the message and inputs it into the AI ​​model.

[0653] Step 5:

[0654] The server's AI model analyzes messages and generates appropriate responses. These responses include natural language processing and speech synthesis.

[0655] Step 6:

[0656] The server sends the generated response to the terminal in JSON format.

[0657] Step 7:

[0658] The device displays or plays the received response to the user.

[0659] Memory Share

[0660] Step 1:

[0661] Users enter text, images, and videos into fields on their device interface to post episodes.

[0662] Step 2:

[0663] The user clicks the submit button.

[0664] Step 3:

[0665] The device sends the posted data to the server in JSON format.

[0666] Step 4:

[0667] The server analyzes the received episodes, converts them to an appropriate format, and stores them in the database.

[0668] Step 5:

[0669] The user enters a question about a specific memory.

[0670] Step 6:

[0671] The server searches the database for relevant episodes and retrieves the corresponding data.

[0672] Step 7:

[0673] The server's AI model generates natural-sounding responses based on the acquired episodes.

[0674] Step 8:

[0675] The server sends the generated response to the terminal in JSON format.

[0676] Step 9:

[0677] The device displays or plays the received response to the user.

[0678] Privacy and security, ethical considerations

[0679] Step 1:

[0680] The server encrypts all incoming information and stores it in the database.

[0681] Step 2:

[0682] Users can request data deletion or changes to their privacy settings.

[0683] Step 3:

[0684] The device sends the user's request to the server in JSON format.

[0685] Step 4:

[0686] The server searches the database for the relevant data and deletes it or updates the privacy settings upon the user's request.

[0687] Step 5:

[0688] The server notifies the terminal in JSON format that the request processing is complete.

[0689] Step 6:

[0690] The device displays the received notification to the user.

[0691] (Example 1)

[0692] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0693] Currently, many users are seeking ways to digitally manage and share memories and information about deceased loved ones, but this is difficult with conventional systems. Furthermore, many systems lack sufficient consideration for privacy and security, making them unsafe to use. Additionally, there is a lack of technology to generate natural conversations based on user input.

[0694] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0695] In this invention, the server includes means for providing an interface to allow the user to input information about a deceased person; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a digital profile of the deceased person based on the stored information; means for generating a response using a generative AI model based on a message sent by the user; means for sending the generated response to the user; means for the server to store the received information using AES-256 encryption technology; means for deleting the relevant information from the database based on a deletion request from the user; and means for updating the user's privacy settings. As a result, the user can safely manage and share information about a deceased person and also enjoy natural conversations through the generative AI model.

[0696] A "user" refers to an individual who uses this system to input, manage, and share information about a deceased person.

[0697] "Interface" refers to the screens and operating methods that users use to access a system and input and manage information.

[0698] A "server" refers to a computer system that analyzes received information, stores it in a database, and generates responses using generative AI models.

[0699] A "database" refers to a digital information management system used to store and manage information about deceased individuals and episodes posted by users.

[0700] A "digital profile" refers to a collection of data that compiles information such as the name, age, photograph, audio sample, and personal anecdotes of a deceased person.

[0701] A "generative AI model" refers to an artificial intelligence model used to analyze user input messages and generate natural-sounding responses.

[0702] "AES-256 encryption technology" refers to a strong encryption technology used to keep received information secure.

[0703] A "deletion request" refers to the operation in which a user sends a request to a server to delete their own data.

[0704] "Privacy settings" refer to the options and actions that allow users to set the scope of visibility and access rights for their data.

[0705] To implement this invention, the system is configured as follows: First, the user inputs basic information about the deceased person using the terminal interface. This interface includes a text field, a file upload function, a voice recording function, and the like.

[0706] Creating a digital profile

[0707] Terminal operation:

[0708] The user opens the terminal interface and enters the deceased person's name, age, photo, audio sample, and personal anecdotes. Once the input is complete, they click the save button. At this point, the terminal converts the entered information into JSON format and sends it to the server as an HTTP request.

[0709] Server processing:

[0710] The server receives an HTTP request, parses the received content, and separates the data into individual fields. Photos are saved as binary data, and audio as audio files. The analyzed data is stored in a database using an RDBMS such as MySQL. This generates a digital profile of the deceased person.

[0711] Specific example:

[0712] The user enters information about "Deceased Person A" and uploads a photo and an audio message. When the user clicks the save button, the device sends this information to the server. The server analyzes the information and saves it to a database, completing the digital profile of "Deceased Person A".

[0713] AI reproduction

[0714] Terminal operation:

[0715] The user opens the terminal interface and types a message to start a conversation into the text box. After typing the message, they click the send button.

[0716] Server processing:

[0717] The terminal converts the message it sends into JSON format and sends it to the server. The server analyzes and processes the received message using a generative AI model (e.g., GPT-3 or ChatGPT). The model generates a natural response, which the server converts back into JSON format and sends back to the terminal. The terminal displays the received response and plays it back as audio if necessary.

[0718] Specific example:

[0719] The user types and sends the message "Hello, Deceased Person A". The server receives this message and uses a generative AI model to generate the response "Hello! How are you?" and sends it to the device. The device displays and plays this response aloud.

[0720] Memory Share

[0721] Terminal operation:

[0722] Users use text fields and file upload functions to enter past memories and episodes. They enter the necessary information and click the submit button.

[0723] Server processing:

[0724] When the device recognizes a click of the post button, it converts the episode information into JSON format and sends it to the server as an HTTP request. The server stores the submitted episode in its database. If the user later initiates a conversation about a particular episode, the server searches and extracts the relevant episode from the database and generates a response using a generative AI model.

[0725] Specific example:

[0726] The user types, "Do you remember the day we went to the beach last summer?" The server searches the database for relevant episodes and generates a response like, "Of course, the beach was really beautiful that day."

[0727] Privacy and security, ethical considerations

[0728] Data encryption:

[0729] The server stores all received information in a database using AES-256 encryption technology. This prevents unauthorized access and data breaches.

[0730] Processing user requests:

[0731] When a user requests data deletion or changes to privacy settings, the device converts this data into JSON format and sends it to the server as an HTTP request. The server receives the request, deletes the data from the database, and updates the settings. If the deletion or update is successful, the server sends a notification back to the device, and the user is shown a message confirming the completion of the deletion.

[0732] Specific example:

[0733] The user requests to delete an episode. The device sends this request to the server, which then deletes the data from the database. A notification is sent to the device upon completion of the deletion, displaying the message "Episode deleted."

[0734] Example of a prompt

[0735] Creating a digital profile: The deceased person's name, age, photo, voice sample, personal anecdotes, etc., are entered on a device and saved to the server's database.

[0736] AI-powered reproduction: The user enters a message, and the server generates a natural response using an AI model.

[0737] Memory sharing: Users post past episodes from their devices, and the server saves them to a database.

[0738] Privacy and Security: The server encrypts all incoming information and processes user data deletion requests.

[0739] In this way, the system provides users with an interactive experience with deceased loved ones, creating opportunities to share memories, while also ensuring a safe and secure system that respects privacy and security.

[0740] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0741] Creating a digital profile

[0742] Step 1:

[0743] The user opens the device's interface and enters the deceased person's name, age, photo, voice sample, personal anecdotes, etc. The entered information is temporarily stored in the device's local memory.

[0744] Input: User enters name, age, photo, voice sample, and episode.

[0745] Output: Information temporarily stored in the terminal's local memory.

[0746] Step 2:

[0747] After the user has entered all the information, they click the save button. The device converts the entered information into JSON format and sends it to the server as an HTTP request.

[0748] Input: Click the save button.

[0749] Output: Information in JSON format sent to the server.

[0750] Step 3:

[0751] The server receives an HTTP request, parses the received content, and divides the data into individual fields. For example, it might save the name as a string field, the photo as binary data, and the audio as an audio file.

[0752] Input: Information sent to the server in JSON format.

[0753] Output: Data for each field analyzed within the server.

[0754] Step 4:

[0755] The server saves the analyzed data to a database using an RDBMS such as MySQL. If the save is successful, the server sends a success message to the terminal in JSON format.

[0756] Input: Data separated into each field.

[0757] Output: Data stored in the database and a success message sent to the terminal.

[0758] AI reproduction

[0759] Step 1:

[0760] The user opens the terminal interface and types a message to start a conversation into the text box. After typing the message, they click the send button.

[0761] Input: User input of a text message.

[0762] Output: Messages temporarily stored within the device.

[0763] Step 2:

[0764] The device recognizes the click of the send button, converts the message to JSON format, and sends it to the server as an HTTP request.

[0765] Input: Click the submit button.

[0766] Output: A JSON-formatted message sent to the server.

[0767] Step 3:

[0768] The server analyzes and processes received messages using a generating AI model (e.g., GPT-3 or ChatGPT). The model then generates a natural-sounding response.

[0769] Input: A JSON-formatted message sent to the server.

[0770] Output: Response generated through the generative AI model.

[0771] Step 4:

[0772] The server converts the generated response into JSON format and sends it to the terminal as an HTTP response. The terminal displays the received response and plays it aloud if necessary.

[0773] Input: The generated response.

[0774] Output: A JSON-formatted response sent to the terminal, along with its display and audio playback on the terminal.

[0775] Memory Share

[0776] Step 1:

[0777] The user opens the terminal interface and uses text fields and file upload functions to enter past memories and episodes. They enter the necessary information and click the submit button.

[0778] Input: User input of episodes and files.

[0779] Output: Episode information temporarily stored on the device.

[0780] Step 2:

[0781] The device recognizes the click of the post button, converts the episode information into JSON format, and sends it to the server as an HTTP request.

[0782] Input: Click the submit button.

[0783] Output: Episode information in JSON format sent to the server.

[0784] Step 3:

[0785] The server saves the received episode to the database. After saving, the server sends a success message to the terminal in JSON format.

[0786] Input: Episode information in JSON format sent to the server.

[0787] Output: Episode information stored in the database and a success message sent to the terminal.

[0788] Step 4:

[0789] If a user later initiates a conversation based on a specific episode, the server searches and extracts the relevant episode from the database and generates a response using a generative AI model.

[0790] Input: User-submitted messages based on their experiences.

[0791] Output: Natural responses generated through a generative AI model.

[0792] Privacy and security, ethical considerations

[0793] Step 1:

[0794] The server stores all received information in a database using AES-256 encryption technology. This prevents unauthorized access and data breaches.

[0795] Input: Information sent to the server.

[0796] Output: Database storage encrypted using AES-256 technology.

[0797] Step 2:

[0798] When a user requests data deletion or changes to privacy settings, the device converts this information into JSON format and sends it to the server as an HTTP request.

[0799] Input: User request for deletion or change of privacy settings.

[0800] Output: A JSON-formatted request sent to the server.

[0801] Step 3:

[0802] The server receives the request, deletes the relevant data from the database, and updates the settings. If the deletion or update is successful, the server sends a success message to the terminal.

[0803] Input: Deletion or privacy setting change request in JSON format.

[0804] Output: Data deleted from the database, or updated privacy settings, and a success message notified to the user.

[0805] (Application Example 1)

[0806] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0807] Currently, many users wish to share memories of deceased family and friends and maintain emotional connections through dialogue. However, existing systems lack the technology to meticulously profile information about deceased individuals and generate natural conversations using AI. Furthermore, concerns about privacy and security necessitate a safe and secure system. This invention aims to solve these problems and provide a system that allows users to engage in natural virtual dialogues with deceased individuals.

[0808] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0809] In this invention, the server includes means for providing an interface to allow the user to input information about a deceased person; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a profile of the deceased person based on the stored information; means for generating a response using a generative AI model based on a message sent by the user; means for sending the generated response to the user; means for encrypting and decrypting the profile data; means for storing past episodes and having the generative AI model generate responses based on these; and means for generating and sending prompt messages to the user's terminal. This enables the user to have a natural conversation with the deceased person and share memories. Furthermore, it enables safe use that takes privacy and security into consideration.

[0810] A "user" is someone who uses the system to input information about a deceased person and then engages in subsequent conversations and shares stories about them.

[0811] An "interface" refers to the screen or control panel that a user uses to operate a device, input information, and transmit it.

[0812] A "profile" is a collection of information that includes the name, age, photograph, audio sample, and personal anecdotes of a deceased person.

[0813] A "generative AI model" is an artificial intelligence model used to generate natural-sounding responses based on user input messages and referencing the profiles of deceased individuals.

[0814] A "response" refers to the reply or reaction from a generative AI model to a message sent by a user.

[0815] A "database" is an electronic information management system used by a server to store information and episodes it receives.

[0816] "Encryption" is the process of transforming data using special techniques to make it unreadable, in order to protect it from unauthorized access.

[0817] "Decryption" is the process of restoring encrypted data to its original, readable state.

[0818] An "episode" is a user-submitted memory or event related to a deceased person.

[0819] A "prompt sentence" is an input sentence for a generative AI model, and it is a sentence necessary to generate an accurate response based on the user's message.

[0820] A "terminal" refers to a user-operated device such as a smartphone, and is used to provide the interface for this system.

[0821] The system necessary to implement this invention is configured as follows: The system mainly consists of a user terminal, a server, a database, and a generative AI model.

[0822] Creating a digital profile

[0823] The user first uses the terminal's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). Specifically, they use the text fields, file upload function, and voice recording function on the terminal's screen. This information is sent to the server when the user clicks the save button. The server analyzes the received information and stores it in a database, generating a digital profile of the deceased person.

[0824] AI reproduction

[0825] The user enters a message to initiate a conversation through the terminal's interface. The server receives the message sent from the terminal and analyzes it using a generative AI model. It then generates a natural response based on the user's digital profile and sends it to the terminal for display to the user.

[0826] Memory Share

[0827] Users use text fields and file upload functions to post past memories and episodes. By entering the necessary information and clicking the submit button, the information is sent to the server. The server stores the received episodes in a database. Subsequently, when a user starts a conversation based on a particular episode, the server searches for previously stored episodes and generates a response using a generative AI model based on the relevant information.

[0828] Privacy and security, ethical considerations

[0829] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data leaks. Furthermore, if a user requests data deletion or changes to privacy settings, the device sends this request to the server, which then deletes the relevant data from the database and updates the settings.

[0830] To give a specific example, when a user types and sends the message "Hello, Taro," the server receives this message, generates a response "Hello! How are you?" through a generative AI model, and sends and displays it on the device.

[0831] Furthermore, if you want to talk about past memories, you can type something like, "Do you remember the day we went to the beach last summer?" The server will search its database for relevant episodes and generate a response like, "Of course, the beach was really beautiful that day."

[0832] Specific examples of hardware and software include smartphones (iOS, Android) as terminals, Flask as a server-side framework, SQLite and PostgreSQL as databases, and GPT-3 (using the Hugging Face transformers library) as a generative AI model.

[0833] Specific examples of prompt messages include user messages such as "What are you doing today?". Questions like "How are you doing?" and messages that evoke past events, such as "Do you remember the day we went to the beach last summer?" are also used as prompt messages.

[0834] This allows users to have natural conversations with deceased loved ones and share memories. Furthermore, it enables safe use with consideration for privacy and security.

[0835] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0836] Step 1:

[0837] The user operates the device's interface to enter basic information about the deceased person. This interface includes text fields, a file upload function, and a voice recording function. Once all the information is entered, the user clicks the save button to confirm the input. Examples of input data include name, age, photo, voice sample, and personal anecdotes.

[0838] Step 2:

[0839] The terminal sends information entered by the user to the server. During this process, the terminal converts the input data into JSON format and transmits it securely using the HTTPS protocol. The data received by the server includes name, age, photo, voice sample, and episode.

[0840] Step 3:

[0841] The server parses the received information and stores it in a database. The server encrypts the data using encryption techniques and later stores it in an appropriate format in each field to maintain data integrity. This data is stored in an SQL database (e.g., PostgreSQL). The input here is user data, and the output is encrypted data stored in the database.

[0842] Step 4:

[0843] The user enters a message to initiate a conversation on the terminal interface. By typing the message in the text box and clicking the send button, the message is sent to the server. Examples of specific prompt messages include "Hello, how are you?".

[0844] Step 5:

[0845] The server analyzes messages received from the terminal. First, it decodes and retrieves the target profile from the database and formats it into an appropriate format for input into a generating AI model (e.g., GPT-3). The input data consists of messages from the user and profile information, and the output is a prompt message provided to the generating AI model.

[0846] Step 6:

[0847] The server uses a generative AI model to generate a natural response from the prompt. The generative AI model generates an appropriate response based on profile information and the user's message. This process results in a response such as, "Hello! How are you?"

[0848] Step 7:

[0849] The server sends the generated response to the user's terminal. The response is encoded in JSON format and sent securely using the HTTPS protocol. The input data is the generated response message, and the output data is the response displayed on the user's terminal.

[0850] Step 8:

[0851] The device displays the response received by the user. This can be done by displaying the message as text in the UI or playing it as audio. This allows the user to continue the conversation.

[0852] Step 9:

[0853] The system uses text fields and file upload functions to allow users to post past memories and episodes. Users confirm their input data with a submit button, and the device sends the data to the server. The data sent includes memories and episodes.

[0854] Step 10:

[0855] The server saves the received episodes to the database. The data is encrypted during this process. The input data is the episode information, and the output data is the encrypted episode information stored in the database.

[0856] Step 11:

[0857] When a user initiates a conversation based on a past episode, the server searches the database for the relevant episode and generates a corresponding response using a generative AI model. For example, if a user types, "Do you remember the day we went to the beach last summer?", the server searches for the relevant episode and generates a response such as, "Of course, the beach was really beautiful that day."

[0858] Step 12:

[0859] The server processes data encryption, decryption, and deletion requests based on the user's privacy and security settings. This prevents unauthorized access and data breaches, ensuring secure use.

[0860] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0861] To carry out this invention, the system consists of the following elements.

[0862] Creating a digital profile

[0863] Terminal operation:

[0864] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities.

[0865] Server processing:

[0866] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. This generates a digital profile of the deceased person.

[0867] Specific example:

[0868] Users input a photo of the deceased "Ichiro Tanaka," along with audio messages from his lifetime and information about events he enjoyed. The server receives the submitted information and stores it in a database as Ichiro Tanaka's digital profile.

[0869] AI reproduction

[0870] Terminal operation:

[0871] The user types a message to start a conversation on the device's interface. The conversation begins when the user clicks the send button.

[0872] Server processing:

[0873] The device sends a message, which is then received. The server then uses an AI model to analyze the received message and generate a natural response based on the user's digital profile. The generated response is sent to the device and displayed to the user.

[0874] Specific example:

[0875] The user types and sends the message "Hello, Ichiro". The server receives this message and, through its AI model, generates the response "Hello! How are you?" and sends it to the device. The device then displays or plays this response aloud.

[0876] Memory Share

[0877] Terminal operation:

[0878] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the submit button.

[0879] Server processing:

[0880] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using an AI model based on the relevant information.

[0881] Specific example:

[0882] The user types, "Do you remember last year's fireworks display?" The server searches the database for relevant episodes and generates a response saying, "Of course, the fireworks that day were really beautiful."

[0883] Emotion recognition by an emotion engine

[0884] Terminal operation:

[0885] When a user enters a text message, an engine works to analyze the emotions behind that message.

[0886] Server processing:

[0887] When receiving a message sent from a device, the emotion engine analyzes the message content and recognizes the emotional state (e.g., joy, sadness, anger). Based on the recognized emotion, the server adjusts the AI ​​model, changing the tone and content of the response.

[0888] Specific example:

[0889] The user sends the message, "I'm really sad today." The server's emotion engine analyzes this message and recognizes that the user is in a state of sadness. Based on this information, the server generates a comforting response, "It's okay, I'll listen," and sends it to the device.

[0890] Privacy and Security

[0891] Data encryption:

[0892] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches.

[0893] Processing user requests:

[0894] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server then deletes the relevant data from the database and updates the settings.

[0895] Specific example:

[0896] The user requests to delete an episode. The device sends this request to the server, which deletes the episode from the database. A notification is sent to the device upon completion of the deletion, and the user is shown "Episode deleted."

[0897] In this way, the system provides users with an interactive experience with their deceased loved ones, creating opportunities to share memories. Furthermore, by utilizing an emotion engine, it can respond in accordance with the user's emotions, resulting in a more personalized experience. With consideration for privacy and security, it provides a system that can be used with peace of mind.

[0898] The following describes the processing flow.

[0899] Creating a digital profile

[0900] Step 1:

[0901] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.).

[0902] Step 2:

[0903] The user completes the input and clicks the save button.

[0904] Step 3:

[0905] The terminal sends the entered information to the server in JSON format.

[0906] Step 4:

[0907] The server analyzes the received information and converts it to an appropriate format, such as Unicode.

[0908] Step 5:

[0909] The server saves the information to the database. After saving is complete, a notification is sent to the device confirming the save is finished.

[0910] Step 6:

[0911] The device receives a notification that saving is complete and displays "Digital profile saved" to the user.

[0912] AI reproduction

[0913] Step 1:

[0914] The user enters a message to start a conversation using the device's interface.

[0915] Step 2:

[0916] The user clicks the submit button.

[0917] Step 3:

[0918] The terminal sends the entered message to the server in JSON format.

[0919] Step 4:

[0920] The server receives the message and inputs it into the AI ​​model.

[0921] Step 5:

[0922] The server's AI model analyzes messages and generates appropriate responses. These responses include natural language processing and speech synthesis.

[0923] Step 6:

[0924] The server sends the generated response to the terminal in JSON format.

[0925] Step 7:

[0926] The device displays or plays the received response to the user via audio.

[0927] Memory Share

[0928] Step 1:

[0929] Users enter text, images, and videos into fields on their device interface to post episodes.

[0930] Step 2:

[0931] The user clicks the submit button.

[0932] Step 3:

[0933] The device sends the posted data to the server in JSON format.

[0934] Step 4:

[0935] The server analyzes the received episodes, converts them to an appropriate format, and stores them in the database.

[0936] Step 5:

[0937] The user enters a question about a specific memory.

[0938] Step 6:

[0939] The server searches the database for relevant episodes and retrieves the corresponding data.

[0940] Step 7:

[0941] The server's AI model generates natural-sounding responses based on the acquired episodes.

[0942] Step 8:

[0943] The server sends the generated response to the terminal in JSON format.

[0944] Step 9:

[0945] The device displays or plays aloud the response it receives to the user.

[0946] Emotion recognition by an emotion engine

[0947] Step 1:

[0948] When a user enters a text message, an engine works to analyze the emotions behind that message.

[0949] Step 2:

[0950] The terminal sends the entered message to the server in JSON format.

[0951] Step 3:

[0952] The server receives the message, and the emotion engine analyzes the message's content to recognize the emotional state (e.g., joy, sadness, anger).

[0953] Step 4:

[0954] Based on the emotions the server recognizes, it adjusts the AI ​​model, changing the tone and content of its responses.

[0955] Step 5:

[0956] The server sends the generated response to the terminal in JSON format.

[0957] Step 6:

[0958] The device displays or plays the received response to the user via audio.

[0959] Privacy and Security

[0960] Step 1:

[0961] The server encrypts all incoming information and stores it in the database.

[0962] Step 2:

[0963] Users can request data deletion or changes to their privacy settings.

[0964] Step 3:

[0965] The device sends the user's request to the server in JSON format.

[0966] Step 4:

[0967] The server searches the database for the relevant data and deletes it or updates the privacy settings upon the user's request.

[0968] Step 5:

[0969] The server notifies the terminal in JSON format that the request processing is complete.

[0970] Step 6:

[0971] Displays notifications received by the device to the user.

[0972] (Example 2)

[0973] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0974] In modern society, there is a need for systems that effectively preserve memories of the deceased and allow users to communicate with them interactively. Natural conversations based on the deceased's emotions and past events are particularly important, while ensuring data privacy and security remains a crucial challenge. Furthermore, technologies that can quickly and accurately analyze user input and generate appropriate responses are essential.

[0975] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0976] In this invention, the server includes means for analyzing messages sent by the user and generating responses using a generative AI model, means for analyzing the emotional state of the user's input message and adjusting the response based on the emotional state, and means for encrypting all received information and storing it in a database. This enables users to communicate with deceased loved ones in an interactive manner, share personal memories and episodes, and use the system with peace of mind.

[0977] A "user" is an individual who uses the system to input information about the deceased or to engage in conversations with them.

[0978] A "deceased person" is someone whose profile and memories are saved in the system, and who has passed away and who can be the subject of virtual conversations with the user.

[0979] An "interface" refers to the user interface or tools that a user uses to input information about a deceased person.

[0980] A "server" is a central computer system that receives information sent by users, processes it, and stores it in a database.

[0981] A "database" is a computer system used to structurally store and manage data such as information and anecdotes about a deceased person.

[0982] A "digital profile" is a collection of data, including the deceased person's name, age, photograph, audio sample, and personal anecdotes, which is stored in a database.

[0983] A "generative AI model" is an artificial intelligence algorithm used to analyze user messages and generate natural-sounding responses.

[0984] "Emotional state" refers to the type and intensity of emotions analyzed from the user's input messages.

[0985] "Encryption" is a technology that transforms data using specific algorithms to protect it from unauthorized access and leakage.

[0986] An "episode" is a detailed description or text of memories or events related to the deceased that the user has entered.

[0987] A "response" is a response message generated by a generative AI model and sent to the user.

[0988] "Privacy settings" are settings that allow users to define the scope of their data's visibility and use within the system.

[0989] This invention relates to a system that allows users to input information about a deceased person and engage in interactive conversations using a generated AI model based on that information. This system mainly consists of a terminal, a server, and a database.

[0990] Creating a digital profile

[0991] Terminal operation:

[0992] The user uses the device's interface to enter basic information about the deceased (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities. For example, the user might enter the deceased's name, "Mr. / Ms. A," a photo, a voice message, and their favorite events.

[0993] Server processing:

[0994] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. For example, the server receives a photo and voice message from "Person A" and stores them in the database to generate Person A's digital profile.

[0995] AI reproduction

[0996] Terminal operation:

[0997] The user enters a message to start a conversation on the terminal's interface. For example, they might type "Hello, A" and click the send button.

[0998] Server processing:

[0999] The server receives a message sent by the terminal. Next, the server analyzes the received message using a generative AI model (e.g., GPT-4) and generates a natural response based on the digital profile. The generated response is sent to the terminal and displayed to the user. For example, if the server receives the message "Hello, A," the generative AI model generates the response "Hello! How are you?" and sends it to the terminal.

[1000] Memory Share

[1001] Terminal operation:

[1002] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the post button. For example, a user might enter text and photos on the theme of "Last year's fireworks display."

[1003] Server processing:

[1004] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using a generative AI model based on the relevant information. For example, upon receiving the message, "Do you remember last year's fireworks display?", the server generates the response, "Of course, the fireworks that day were really beautiful."

[1005] Emotion recognition by an emotion engine

[1006] Terminal operation:

[1007] When a user types a text message, an engine works to analyze the emotion behind that message. For example, the user might type, "I'm really sad today."

[1008] Server processing:

[1009] When receiving a message sent from a device, the emotion engine analyzes the message content and recognizes the emotional state (e.g., joy, sadness, anger). Based on the recognized emotion, the server adjusts the generative AI model to change the tone and content of the response. For example, in response to a message that says, "I'm really sad today," the server might generate a comforting response such as, "It's okay, I'll listen."

[1010] Privacy and Security

[1011] Data encryption:

[1012] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches. For example, all transmitted data is encrypted using the TLS / SSL protocol.

[1013] Processing user requests:

[1014] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server deletes the relevant data from the database and updates the settings. For example, if a user requests to delete a specific episode, the server deletes that episode from the database and sends a message to the device stating, "Episode deleted."

[1015] This system allows users to enjoy interactive experiences with deceased loved ones and share personal memories. Furthermore, by utilizing an emotion engine and generative AI models, it provides personalized responses tailored to the user's emotions. In addition, the system is designed with privacy and security in mind, ensuring safe use.

[1016] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1017] Creating a digital profile

[1018] Terminal operation:

[1019] Step 1:

[1020] The user opens the device's interface. The user enters the deceased's name and age in the text fields on the interface, adds a photo using the file upload function, and records a voice sample using the voice recording function. This becomes the input data.

[1021] Server processing:

[1022] Step 2:

[1023] Once the user completes the information entry and clicks the save button, the device sends the entered information (text, photos, audio) to the server in JSON format. This is the input process.

[1024] Step 3:

[1025] The server parses the received JSON data and extracts each field (name, age, photo, audio, etc.). This parsing process prepares the data structure for storage in the database. Next, the server saves the parsed data to the database in the appropriate format. Specifically, it saves the name and age as text data, and the photo and audio as binary data. This is the output.

[1026] AI reproduction

[1027] Terminal operation:

[1028] Step 1:

[1029] The user opens the chat interface on their device and enters a message such as "Hello, A" into the text field. This is the input data.

[1030] Step 2:

[1031] When the user clicks the send button, the terminal sends the entered message to the server in JSON format. This is the input.

[1032] Server processing:

[1033] Step 3:

[1034] The server analyzes the received message and compares it to a digital profile. Based on the comparison results, the server uses a generative AI model (e.g., GPT-4) to create a prompt for generating an appropriate response. This prompt might take the form of, for example, "The user said 'Hello, Mr. / Ms. A' to the deceased. Please respond as the deceased." This is data processing.

[1035] Step 4:

[1036] The generated prompt is used to run the generative AI model, which then produces a natural-sounding response (e.g., "Hello! How are you?"). This is the output.

[1037] Step 5:

[1038] The server sends the generated response to the terminal in JSON format. This is also output.

[1039] Terminal operation:

[1040] Step 6:

[1041] The terminal displays the received response in the chat interface for the user to see. It also plays it back as audio if necessary. This is the final output.

[1042] Memory Share

[1043] Terminal operation:

[1044] Step 1:

[1045] Users input past episodes and memories using text fields or file upload functions. For example, they might input an episode titled "Last year's fireworks display" along with a photo. This is the input data.

[1046] Step 2:

[1047] When a user clicks the "Submit" button, the device sends the entered episode data to the server in JSON format. This is the input.

[1048] Server processing:

[1049] Step 3:

[1050] The server analyzes the received episode data and stores it in the database. The analysis process extracts the episode text and photo data, which are then inserted into the appropriate records in the database. This is the output.

[1051] Step 4:

[1052] When a user sends the message "Do you remember last year's fireworks display?" to the server, the server searches its database for relevant episodes. Specifically, it searches using the keyword "fireworks display" and extracts the corresponding data. This is the input data search.

[1053] Step 5:

[1054] Based on the episode information extracted by the server, a generative AI model is used to generate a response such as, "Of course, the fireworks that day were really beautiful." This is data processing.

[1055] Step 6:

[1056] The server sends the generated response to the terminal, which then displays it to the user. This is the final output.

[1057] Emotion recognition by an emotion engine

[1058] Terminal operation:

[1059] Step 1:

[1060] The user types a text message and sends it saying, "I'm really sad today." This is the input data.

[1061] Step 2:

[1062] When the user clicks the send button, the device sends the message to the server in JSON format. This is the input.

[1063] Server processing:

[1064] Step 3:

[1065] The server analyzes the received message and uses an emotion engine to recognize the user's emotional state as "sad." This emotion analysis process is a data calculation.

[1066] Step 4:

[1067] After the emotional state is recognized, the server adjusts the generating AI model to produce a comforting response such as, "It's okay, I'll listen." This is the output.

[1068] Step 5:

[1069] The server sends the generated response to the terminal, which then displays it to the user. It also plays it back as audio if necessary. This is the final output.

[1070] Privacy and Security

[1071] Data encryption:

[1072] Step 1:

[1073] The server encrypts all incoming information. Specifically, it encrypts the information using the SSL / TLS protocol and stores it in the database. This is the process of protecting the input data.

[1074] Processing user requests:

[1075] Step 2:

[1076] The user requests data deletion or changes to privacy settings from their device. For example, they might request, "I want to delete a specific episode." This is the input data.

[1077] Step 3:

[1078] The terminal sends a request to the server. This is the input.

[1079] Server processing:

[1080] Step 4:

[1081] The server analyzes the received request and deletes the corresponding data from the database. Specifically, it searches for the relevant episode based on the episode ID included in the request and physically deletes it from the database. This is data processing and output.

[1082] Step 5:

[1083] After the deletion is complete, the server notifies the terminal of the result. A message "Episode deleted" is generated and sent to the terminal. This is the final output.

[1084] This system provides users with an interactive experience with their deceased loved ones, creating opportunities to share memories. By utilizing an emotion engine and generative AI models, it can respond to the user's emotions, resulting in a more personalized experience. Furthermore, privacy and security measures are in place, ensuring safe and secure use.

[1085] (Application Example 2)

[1086] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1087] Traditional interactive systems for sharing memories of deceased loved ones lacked the ability to generate responses that took into account the user's emotions and location. This made it difficult to provide information about the deceased and their past in a more natural and personalized way. Furthermore, ensuring the privacy and security of information was insufficient. As a result, users were unable to enjoy a high level of immersion and security.

[1088] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for providing an interface to allow the user to input information about the deceased; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a digital profile of the deceased based on the stored information; means for generating a response using a generation AI model based on a message sent by the user; means for acquiring information at a specific spot using the customer's location information and generating a response; and means for transmitting the generated response to the user and playing it back by display or audio. This makes it possible to provide natural interaction tailored to the user's individual circumstances and location, ensuring the privacy and security of information while allowing for a higher level of immersion.

[1089] 1. A "user" is a person who uses this system to input, transmit, and manage information about the deceased.

[1090] 2. "Interface" refers to the screens, fields, and means of operating them that are necessary for a user to input information about the deceased.

[1091] 3. A "server" is a central computer system that processes information received from users and stores it in a database.

[1092] 4. A "database" is a data storage system used by a server to store, search, and manage information it receives.

[1093] 5. A "digital profile" refers to virtual personal data that includes basic information and memories of the deceased.

[1094] 6. A "generative AI model" is an artificial intelligence algorithm that analyzes messages sent by users and generates appropriate responses.

[1095] 7. A "response" is the answer or response that a server generates in response to user input or actions.

[1096] 8. "Location information" refers to GPS data that indicates the user's current location.

[1097] 9. A "spot" is a location associated with specific information or a particular response.

[1098] 10. "Audio playback" refers to a method of delivering a message generated as a response to the user as audio.

[1099] 11. An "episode" is a specific story or memory shared by a user about a deceased person or past events.

[1100] 12. A "message" is a piece of text or a request that a user inputs through the interface and sends to the server.

[1101] 13. "Sentiment analysis" refers to the process of recognizing a user's emotional state from their messages and statements.

[1102] 14. "Personalization" means providing customized support tailored to each individual user.

[1103] 15. "Privacy settings" are settings that allow users to decide how their information is shared and how their data is handled.

[1104] 16. "Encryption" is a technique that converts data into a format that cannot be read in order to securely protect it.

[1105] 17. The "episode submission field" is an input field where users can enter past events or memories.

[1106] The system for carrying out this invention consists of the following hardware and software.

[1107] Hardware and software used

[1108] Hardware: Smartphones, smart glasses, head-mounted displays

[1109] Software: AWS (Amazon Web Services), MySQL, Python, TensorFlow, OpenCV

[1110] System Configuration

[1111] The system primarily consists of user terminals, servers, and databases. The operation and role of each major component are described in detail below.

[1112] User terminal

[1113] The user terminal uses devices such as smartphones, smart glasses, and head-mounted displays. The user terminal provides an interface for inputting information about the deceased and functions for transmitting that information. This allows users to easily input data and send it to the server.

[1114] server

[1115] The server processes information received from the user and generates a digital profile. It also uses a generative AI model to produce natural-sounding responses based on the user's input messages. The server stores the received information in a MySQL database and searches this data as needed to generate the best possible answers.

[1116] database

[1117] The database plays a role in securely storing information received from the server. The database contains encrypted data such as information about deceased individuals, user-submitted episodes, and past message histories.

[1118] Processing flow

[1119] The user first uses an interface to enter information about the deceased. This interface includes text fields, a file upload function, and a voice recording function. The information entered by the user is sent to the server, analyzed, and stored in a database. This generates a digital profile of the deceased.

[1120] When a user stands at a specific location, the server uses GPS information from their smartphone or smart glasses to retrieve specific information. Based on this, a generative AI model generates a relevant response and sends it to the user's device. The user's device then displays or plays this response as text or audio.

[1121] Emotion analysis and personalization

[1122] When a user enters a message, the sentiment analysis engine activates to recognize the user's emotions. Based on these emotions, the generative AI model generates a personalized response. For example, if a user enters "I'm really sad today," the server performs sentiment analysis and, depending on the user's emotional state, generates a comforting response such as "It's okay, I'll listen" and sends it to the user's device.

[1123] Privacy and Security

[1124] All data is end-to-end encrypted and stored in the database. If a user requests data deletion or changes to privacy settings, the server responds immediately, deleting the relevant data from the database or updating the settings. This ensures users can use the system with peace of mind.

[1125] Examples of specific cases and prompt statements

[1126] As a concrete example, if a profile of a former regular customer named "Ichiro Tanaka" is registered, and a customer at the store says using smart glasses, "I've found Ichiro's usual seat," the AI ​​will respond, "Really? Ichiro used to drink coffee here often."

[1127] Example of a prompt:

[1128] "Tell me about this place."

[1129] "Tell me about Christmas events from the past."

[1130] "Please share your memories from here."

[1131] In this way, the system provides users with a personalized, interactive experience, deepening their connection with deceased loved ones and past events. Furthermore, security measures ensure that users can use the system with peace of mind.

[1132] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1133] Step 1:

[1134] The user uses the terminal interface to enter basic information about the deceased (name, age, photo, audio sample, personal anecdotes, etc.). The entered information is sent from the user's terminal to the server. Specifically, the user enters information into a text field and clicks the "Send" button. The input is text or multimedia files (photo, audio), and the output is a data packet sent to the server.

[1135] Step 2:

[1136] The server parses the received information and stores it in a MySQL database. The server converts the input data into the appropriate format and inserts it into the database using SQL statements. The input is the user's data sent, and the output is a digital profile stored in the database.

[1137] Step 3:

[1138] When a user stands at a specific location, the server uses the device's GPS information to obtain their location. The device sends current coordinate data to the server, which then uses this data to search for relevant information. The input is the GPS data sent from the device, and the output is the relevant information returned by the server.

[1139] Step 4:

[1140] The server generates a response using a generative AI model based on relevant information. Specifically, the server searches for relevant episodes in the database and inputs them as prompts into the generative AI model. The server then analyzes the text generated by the model and produces an appropriate response. The input is relevant information based on location data, and the output is the generated response.

[1141] Step 5:

[1142] The server sends the generated response to the user's terminal, which then displays or plays it as text or audio. Specifically, the server sends the generated response to the terminal, and the terminal presents the received data to the user in an appropriate format. The input is the generated response, and the output is the text display or audio playback to the user.

[1143] Step 6:

[1144] When a user enters and sends an additional message, the server receives it and uses its sentiment analysis engine to recognize the emotion. Based on the analysis results, the server adjusts its AI model to generate a response appropriate to the emotion. The input is the user's message, and the output is the emotion-appropriate response.

[1145] Step 7:

[1146] The server encrypts all data and securely stores it in the database. The input is the data to be stored, and the output is the encrypted data. When a user requests data deletion or changes to privacy settings, the server reflects this in the database and updates the settings. Specifically, it parses the user's request and deletes or updates the corresponding data from the database. The input is the user's request, and the output is the updated database state.

[1147] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1148] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1149] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1150] [Third Embodiment]

[1151] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1152] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1153] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1154] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1155] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1156] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1157] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1158] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1159] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1160] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1161] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1162] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1163] To implement this invention, the system is configured as follows.

[1164] Creating a digital profile

[1165] Terminal operation:

[1166] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities.

[1167] Server processing:

[1168] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. This generates a digital profile of the deceased person.

[1169] Specific example:

[1170] Users enter a photo of the deceased "Taro Yamada," an audio message from him, and information about events he enjoyed. The server receives the submitted information and stores it in a database as Taro Yamada's digital profile.

[1171] AI reproduction

[1172] Terminal operation:

[1173] The user uses a text box on the device interface to enter a message to start a conversation. They enter the message and click the send button to begin the conversation.

[1174] Server processing:

[1175] The device sends a message, which is then received. The server then uses an AI model to analyze the received message and generate a natural response based on the user's digital profile. The generated response is sent to the device and displayed to the user.

[1176] Specific example:

[1177] The user types and sends the message "Hello, Taro." The server receives this message and, through its AI model, generates the response "Hello! How are you?" which is then sent to the device. The device then displays or plays this response aloud.

[1178] Memory Share

[1179] Terminal operation:

[1180] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the submit button.

[1181] Server processing:

[1182] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using an AI model based on the relevant information.

[1183] Specific example:

[1184] The user types, "Do you remember the day we went to the beach last summer?" The server searches the database for relevant episodes and generates a response saying, "Of course, the beach was really beautiful that day."

[1185] Privacy and security, ethical considerations

[1186] Data encryption:

[1187] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches.

[1188] Processing user requests:

[1189] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server then deletes the relevant data from its database and updates the settings.

[1190] Specific example:

[1191] The user requests to delete an episode. The device sends this request to the server, which deletes the episode from the database. A notification is sent to the device upon completion of the deletion, and the user is shown "Episode deleted."

[1192] In this way, the system provides users with an interactive experience with deceased loved ones and creates opportunities to share memories. It also prioritizes privacy and security, ensuring a system that can be used with peace of mind.

[1193] The following describes the processing flow.

[1194] Creating a digital profile

[1195] Step 1:

[1196] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.).

[1197] Step 2:

[1198] The user completes the input and clicks the save button.

[1199] Step 3:

[1200] The terminal sends the entered information to the server in JSON format.

[1201] Step 4:

[1202] The server analyzes the received information and converts it to an appropriate format, such as Unicode.

[1203] Step 5:

[1204] The server saves the information to the database. After saving is complete, a notification is sent to the device confirming the save is finished.

[1205] Step 6:

[1206] The device receives a notification that saving is complete and displays "Digital profile saved" to the user.

[1207] AI reproduction

[1208] Step 1:

[1209] The user enters a message to start a conversation using the device's interface.

[1210] Step 2:

[1211] The user clicks the submit button.

[1212] Step 3:

[1213] The terminal sends the entered message to the server in JSON format.

[1214] Step 4:

[1215] The server receives the message and inputs it into the AI ​​model.

[1216] Step 5:

[1217] The server's AI model analyzes messages and generates appropriate responses. These responses include natural language processing and speech synthesis.

[1218] Step 6:

[1219] The server sends the generated response to the terminal in JSON format.

[1220] Step 7:

[1221] The device displays or plays the received response to the user.

[1222] Memory Share

[1223] Step 1:

[1224] Users enter text, images, and videos into fields on their device interface to post episodes.

[1225] Step 2:

[1226] The user clicks the submit button.

[1227] Step 3:

[1228] The device sends the posted data to the server in JSON format.

[1229] Step 4:

[1230] The server analyzes the received episodes, converts them to an appropriate format, and stores them in the database.

[1231] Step 5:

[1232] The user enters a question about a specific memory.

[1233] Step 6:

[1234] The server searches the database for relevant episodes and retrieves the corresponding data.

[1235] Step 7:

[1236] The server's AI model generates natural-sounding responses based on the acquired episodes.

[1237] Step 8:

[1238] The server sends the generated response to the terminal in JSON format.

[1239] Step 9:

[1240] The device displays or plays the received response to the user.

[1241] Privacy and security, ethical considerations

[1242] Step 1:

[1243] The server encrypts all incoming information and stores it in the database.

[1244] Step 2:

[1245] Users can request data deletion or changes to their privacy settings.

[1246] Step 3:

[1247] The device sends the user's request to the server in JSON format.

[1248] Step 4:

[1249] The server searches the database for the relevant data and deletes it or updates the privacy settings upon the user's request.

[1250] Step 5:

[1251] The server notifies the terminal in JSON format that the request processing is complete.

[1252] Step 6:

[1253] The device displays the received notification to the user.

[1254] (Example 1)

[1255] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1256] Currently, many users are seeking ways to digitally manage and share memories and information about deceased loved ones, but this is difficult with conventional systems. Furthermore, many systems lack sufficient consideration for privacy and security, making them unsafe to use. Additionally, there is a lack of technology to generate natural conversations based on user input.

[1257] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1258] In this invention, the server includes means for providing an interface to allow the user to input information about a deceased person; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a digital profile of the deceased person based on the stored information; means for generating a response using a generative AI model based on a message sent by the user; means for sending the generated response to the user; means for the server to store the received information using AES-256 encryption technology; means for deleting the relevant information from the database based on a deletion request from the user; and means for updating the user's privacy settings. As a result, the user can safely manage and share information about a deceased person and also enjoy natural conversations through the generative AI model.

[1259] A "user" refers to an individual who uses this system to input, manage, and share information about a deceased person.

[1260] "Interface" refers to the screens and operating methods that users use to access a system and input and manage information.

[1261] A "server" refers to a computer system that analyzes received information, stores it in a database, and generates responses using generative AI models.

[1262] A "database" refers to a digital information management system used to store and manage information about deceased individuals and episodes posted by users.

[1263] A "digital profile" refers to a collection of data that compiles information such as the name, age, photograph, audio sample, and personal anecdotes of a deceased person.

[1264] A "generative AI model" refers to an artificial intelligence model used to analyze user input messages and generate natural-sounding responses.

[1265] "AES-256 encryption technology" refers to a strong encryption technology used to keep received information secure.

[1266] A "deletion request" refers to the operation in which a user sends a request to a server to delete their own data.

[1267] "Privacy settings" refer to the options and actions that allow users to set the scope of visibility and access rights for their data.

[1268] To implement this invention, the system is configured as follows: First, the user inputs basic information about the deceased person using the terminal interface. This interface includes a text field, a file upload function, a voice recording function, and the like.

[1269] Creating a digital profile

[1270] Terminal operation:

[1271] The user opens the terminal interface and enters the deceased person's name, age, photo, audio sample, and personal anecdotes. Once the input is complete, they click the save button. At this point, the terminal converts the entered information into JSON format and sends it to the server as an HTTP request.

[1272] Server processing:

[1273] The server receives an HTTP request, parses the received content, and separates the data into individual fields. Photos are saved as binary data, and audio as audio files. The analyzed data is stored in a database using an RDBMS such as MySQL. This generates a digital profile of the deceased person.

[1274] Specific example:

[1275] The user enters information about "Deceased Person A" and uploads a photo and an audio message. When the user clicks the save button, the device sends this information to the server. The server analyzes the information and saves it to a database, completing the digital profile of "Deceased Person A".

[1276] AI reproduction

[1277] Terminal operation:

[1278] The user opens the terminal interface and types a message to start a conversation into the text box. After typing the message, they click the send button.

[1279] Server processing:

[1280] The terminal converts the message it sends into JSON format and sends it to the server. The server analyzes and processes the received message using a generative AI model (e.g., GPT-3 or ChatGPT). The model generates a natural response, which the server converts back into JSON format and sends back to the terminal. The terminal displays the received response and plays it back as audio if necessary.

[1281] Specific example:

[1282] The user types and sends the message "Hello, Deceased Person A". The server receives this message and uses a generative AI model to generate the response "Hello! How are you?" and sends it to the device. The device displays and plays this response aloud.

[1283] Memory Share

[1284] Terminal operation:

[1285] Users use text fields and file upload functions to enter past memories and episodes. They enter the necessary information and click the submit button.

[1286] Server processing:

[1287] When the device recognizes a click of the post button, it converts the episode information into JSON format and sends it to the server as an HTTP request. The server stores the submitted episode in its database. If the user later initiates a conversation about a particular episode, the server searches and extracts the relevant episode from the database and generates a response using a generative AI model.

[1288] Specific example:

[1289] The user types, "Do you remember the day we went to the beach last summer?" The server searches the database for relevant episodes and generates a response like, "Of course, the beach was really beautiful that day."

[1290] Privacy and security, ethical considerations

[1291] Data encryption:

[1292] The server stores all received information in a database using AES-256 encryption technology. This prevents unauthorized access and data breaches.

[1293] Processing user requests:

[1294] When a user requests data deletion or changes to privacy settings, the device converts this data into JSON format and sends it to the server as an HTTP request. The server receives the request, deletes the data from the database, and updates the settings. If the deletion or update is successful, the server sends a notification back to the device, and the user is shown a message confirming the completion of the deletion.

[1295] Specific example:

[1296] The user requests to delete an episode. The device sends this request to the server, which then deletes the data from the database. A notification is sent to the device upon completion of the deletion, displaying the message "Episode deleted."

[1297] Example of a prompt

[1298] Creating a digital profile: The deceased person's name, age, photo, voice sample, personal anecdotes, etc., are entered on a device and saved to the server's database.

[1299] AI-powered reproduction: The user enters a message, and the server generates a natural response using an AI model.

[1300] Memory sharing: Users post past episodes from their devices, and the server saves them to a database.

[1301] Privacy and Security: The server encrypts all incoming information and processes user data deletion requests.

[1302] In this way, the system provides users with an interactive experience with deceased loved ones, creating opportunities to share memories, while also ensuring a safe and secure system that respects privacy and security.

[1303] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1304] Creating a digital profile

[1305] Step 1:

[1306] The user opens the device's interface and enters the deceased person's name, age, photo, voice sample, personal anecdotes, etc. The entered information is temporarily stored in the device's local memory.

[1307] Input: User enters name, age, photo, voice sample, and episode.

[1308] Output: Information temporarily stored in the terminal's local memory.

[1309] Step 2:

[1310] After the user has entered all the information, they click the save button. The device converts the entered information into JSON format and sends it to the server as an HTTP request.

[1311] Input: Click the save button.

[1312] Output: Information in JSON format sent to the server.

[1313] Step 3:

[1314] The server receives an HTTP request, parses the received content, and divides the data into individual fields. For example, it might save the name as a string field, the photo as binary data, and the audio as an audio file.

[1315] Input: Information sent to the server in JSON format.

[1316] Output: Data for each field analyzed within the server.

[1317] Step 4:

[1318] The server saves the analyzed data to a database using an RDBMS such as MySQL. If the save is successful, the server sends a success message to the terminal in JSON format.

[1319] Input: Data separated into each field.

[1320] Output: Data stored in the database and a success message sent to the terminal.

[1321] AI reproduction

[1322] Step 1:

[1323] The user opens the terminal interface and types a message to start a conversation into the text box. After typing the message, they click the send button.

[1324] Input: User input of a text message.

[1325] Output: Messages temporarily stored within the device.

[1326] Step 2:

[1327] The device recognizes the click of the send button, converts the message to JSON format, and sends it to the server as an HTTP request.

[1328] Input: Click the submit button.

[1329] Output: A JSON-formatted message sent to the server.

[1330] Step 3:

[1331] The server analyzes and processes received messages using a generating AI model (e.g., GPT-3 or ChatGPT). The model then generates a natural-sounding response.

[1332] Input: A JSON-formatted message sent to the server.

[1333] Output: Response generated through the generative AI model.

[1334] Step 4:

[1335] The server converts the generated response into JSON format and sends it to the terminal as an HTTP response. The terminal displays the received response and plays it aloud if necessary.

[1336] Input: The generated response.

[1337] Output: A JSON-formatted response sent to the terminal, along with its display and audio playback on the terminal.

[1338] Memory Share

[1339] Step 1:

[1340] The user opens the terminal interface and uses text fields and file upload functions to enter past memories and episodes. They enter the necessary information and click the submit button.

[1341] Input: User input of episodes and files.

[1342] Output: Episode information temporarily stored on the device.

[1343] Step 2:

[1344] The device recognizes the click of the post button, converts the episode information into JSON format, and sends it to the server as an HTTP request.

[1345] Input: Click the submit button.

[1346] Output: Episode information in JSON format sent to the server.

[1347] Step 3:

[1348] The server saves the received episode to the database. After saving, the server sends a success message to the terminal in JSON format.

[1349] Input: Episode information in JSON format sent to the server.

[1350] Output: Episode information stored in the database and a success message sent to the terminal.

[1351] Step 4:

[1352] If a user later initiates a conversation based on a specific episode, the server searches and extracts the relevant episode from the database and generates a response using a generative AI model.

[1353] Input: User-submitted messages based on their experiences.

[1354] Output: Natural responses generated through a generative AI model.

[1355] Privacy and security, ethical considerations

[1356] Step 1:

[1357] The server stores all received information in a database using AES-256 encryption technology. This prevents unauthorized access and data breaches.

[1358] Input: Information sent to the server.

[1359] Output: Database storage encrypted using AES-256 technology.

[1360] Step 2:

[1361] When a user requests data deletion or changes to privacy settings, the device converts this information into JSON format and sends it to the server as an HTTP request.

[1362] Input: User request for deletion or change of privacy settings.

[1363] Output: A JSON-formatted request sent to the server.

[1364] Step 3:

[1365] The server receives the request, deletes the relevant data from the database, and updates the settings. If the deletion or update is successful, the server sends a success message to the terminal.

[1366] Input: Deletion or privacy setting change request in JSON format.

[1367] Output: Data deleted from the database, or updated privacy settings, and a success message notified to the user.

[1368] (Application Example 1)

[1369] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1370] Currently, many users wish to share memories of deceased family and friends and maintain emotional connections through dialogue. However, existing systems lack the technology to meticulously profile information about deceased individuals and generate natural conversations using AI. Furthermore, concerns about privacy and security necessitate a safe and secure system. This invention aims to solve these problems and provide a system that allows users to engage in natural virtual dialogues with deceased individuals.

[1371] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1372] In this invention, the server includes means for providing an interface to allow the user to input information about a deceased person; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a profile of the deceased person based on the stored information; means for generating a response using a generative AI model based on a message sent by the user; means for sending the generated response to the user; means for encrypting and decrypting the profile data; means for storing past episodes and having the generative AI model generate responses based on these; and means for generating and sending prompt messages to the user's terminal. This enables the user to have a natural conversation with the deceased person and share memories. Furthermore, it enables safe use that takes privacy and security into consideration.

[1373] A "user" is someone who uses the system to input information about a deceased person and then engages in subsequent conversations and shares stories about them.

[1374] An "interface" refers to the screen or control panel that a user uses to operate a device, input information, and transmit it.

[1375] A "profile" is a collection of information that includes the name, age, photograph, audio sample, and personal anecdotes of a deceased person.

[1376] A "generative AI model" is an artificial intelligence model used to generate natural-sounding responses based on user input messages and referencing the profiles of deceased individuals.

[1377] A "response" refers to the reply or reaction from a generative AI model to a message sent by a user.

[1378] A "database" is an electronic information management system used by a server to store information and episodes it receives.

[1379] "Encryption" is the process of transforming data using special techniques to make it unreadable, in order to protect it from unauthorized access.

[1380] "Decryption" is the process of restoring encrypted data to its original, readable state.

[1381] An "episode" is a user-submitted memory or event related to a deceased person.

[1382] A "prompt sentence" is an input sentence for a generative AI model, and it is a sentence necessary to generate an accurate response based on the user's message.

[1383] A "terminal" refers to a user-operated device such as a smartphone, and is used to provide the interface for this system.

[1384] The system necessary to implement this invention is configured as follows: The system mainly consists of a user terminal, a server, a database, and a generative AI model.

[1385] Creating a digital profile

[1386] The user first uses the terminal's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). Specifically, they use the text fields, file upload function, and voice recording function on the terminal's screen. This information is sent to the server when the user clicks the save button. The server analyzes the received information and stores it in a database, generating a digital profile of the deceased person.

[1387] AI reproduction

[1388] The user enters a message to initiate a conversation through the terminal's interface. The server receives the message sent from the terminal and analyzes it using a generative AI model. It then generates a natural response based on the user's digital profile and sends it to the terminal for display to the user.

[1389] Memory Share

[1390] Users use text fields and file upload functions to post past memories and episodes. By entering the necessary information and clicking the submit button, the information is sent to the server. The server stores the received episodes in a database. Subsequently, when a user starts a conversation based on a particular episode, the server searches for previously stored episodes and generates a response using a generative AI model based on the relevant information.

[1391] Privacy and security, ethical considerations

[1392] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data leaks. Furthermore, if a user requests data deletion or changes to privacy settings, the device sends this request to the server, which then deletes the relevant data from the database and updates the settings.

[1393] To give a specific example, when a user types and sends the message "Hello, Taro," the server receives this message, generates a response "Hello! How are you?" through a generative AI model, and sends and displays it on the device.

[1394] Furthermore, if you want to talk about past memories, you can type something like, "Do you remember the day we went to the beach last summer?" The server will search its database for relevant episodes and generate a response like, "Of course, the beach was really beautiful that day."

[1395] Specific examples of hardware and software include smartphones (iOS, Android) as terminals, Flask as a server-side framework, SQLite and PostgreSQL as databases, and GPT-3 (using the Hugging Face transformers library) as a generative AI model.

[1396] Specific examples of prompt messages include user messages such as "What are you doing today?". Questions like "How are you doing?" and messages that evoke past events, such as "Do you remember the day we went to the beach last summer?" are also used as prompt messages.

[1397] This allows users to have natural conversations with deceased loved ones and share memories. Furthermore, it enables safe use with consideration for privacy and security.

[1398] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1399] Step 1:

[1400] The user operates the device's interface to enter basic information about the deceased person. This interface includes text fields, a file upload function, and a voice recording function. Once all the information is entered, the user clicks the save button to confirm the input. Examples of input data include name, age, photo, voice sample, and personal anecdotes.

[1401] Step 2:

[1402] The terminal sends information entered by the user to the server. During this process, the terminal converts the input data into JSON format and transmits it securely using the HTTPS protocol. The data received by the server includes name, age, photo, voice sample, and episode.

[1403] Step 3:

[1404] The server parses the received information and stores it in a database. The server encrypts the data using encryption techniques and later stores it in an appropriate format in each field to maintain data integrity. This data is stored in an SQL database (e.g., PostgreSQL). The input here is user data, and the output is encrypted data stored in the database.

[1405] Step 4:

[1406] The user enters a message to initiate a conversation on the terminal interface. By typing the message in the text box and clicking the send button, the message is sent to the server. Examples of specific prompt messages include "Hello, how are you?".

[1407] Step 5:

[1408] The server analyzes messages received from the terminal. First, it decodes and retrieves the target profile from the database and formats it into an appropriate format for input into a generating AI model (e.g., GPT-3). The input data consists of messages from the user and profile information, and the output is a prompt message provided to the generating AI model.

[1409] Step 6:

[1410] The server uses a generative AI model to generate a natural response from the prompt. The generative AI model generates an appropriate response based on profile information and the user's message. This process results in a response such as, "Hello! How are you?"

[1411] Step 7:

[1412] The server sends the generated response to the user's terminal. The response is encoded in JSON format and sent securely using the HTTPS protocol. The input data is the generated response message, and the output data is the response displayed on the user's terminal.

[1413] Step 8:

[1414] The device displays the response received by the user. This can be done by displaying the message as text in the UI or playing it as audio. This allows the user to continue the conversation.

[1415] Step 9:

[1416] The system uses text fields and file upload functions to allow users to post past memories and episodes. Users confirm their input data with a submit button, and the device sends the data to the server. The data sent includes memories and episodes.

[1417] Step 10:

[1418] The server saves the received episodes to the database. The data is encrypted during this process. The input data is the episode information, and the output data is the encrypted episode information stored in the database.

[1419] Step 11:

[1420] When a user initiates a conversation based on a past episode, the server searches the database for the relevant episode and generates a corresponding response using a generative AI model. For example, if a user types, "Do you remember the day we went to the beach last summer?", the server searches for the relevant episode and generates a response such as, "Of course, the beach was really beautiful that day."

[1421] Step 12:

[1422] The server processes data encryption, decryption, and deletion requests based on the user's privacy and security settings. This prevents unauthorized access and data breaches, ensuring secure use.

[1423] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1424] To carry out this invention, the system consists of the following elements.

[1425] Creating a digital profile

[1426] Terminal operation:

[1427] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities.

[1428] Server processing:

[1429] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. This generates a digital profile of the deceased person.

[1430] Specific example:

[1431] Users input a photo of the deceased "Ichiro Tanaka," along with audio messages from his lifetime and information about events he enjoyed. The server receives the submitted information and stores it in a database as Ichiro Tanaka's digital profile.

[1432] AI reproduction

[1433] Terminal operation:

[1434] The user types a message to start a conversation on the device's interface. The conversation begins when the user clicks the send button.

[1435] Server processing:

[1436] The device sends a message, which is then received. The server then uses an AI model to analyze the received message and generate a natural response based on the user's digital profile. The generated response is sent to the device and displayed to the user.

[1437] Specific example:

[1438] The user types and sends the message "Hello, Ichiro". The server receives this message and, through its AI model, generates the response "Hello! How are you?" and sends it to the device. The device then displays or plays this response aloud.

[1439] Memory Share

[1440] Terminal operation:

[1441] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the submit button.

[1442] Server processing:

[1443] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using an AI model based on the relevant information.

[1444] Specific example:

[1445] The user types, "Do you remember last year's fireworks display?" The server searches the database for relevant episodes and generates a response saying, "Of course, the fireworks that day were really beautiful."

[1446] Emotion recognition by an emotion engine

[1447] Terminal operation:

[1448] When a user enters a text message, an engine works to analyze the emotions behind that message.

[1449] Server processing:

[1450] When receiving a message sent from a device, the emotion engine analyzes the message content and recognizes the emotional state (e.g., joy, sadness, anger). Based on the recognized emotion, the server adjusts the AI ​​model, changing the tone and content of the response.

[1451] Specific example:

[1452] The user sends the message, "I'm really sad today." The server's emotion engine analyzes this message and recognizes that the user is in a state of sadness. Based on this information, the server generates a comforting response, "It's okay, I'll listen," and sends it to the device.

[1453] Privacy and Security

[1454] Data encryption:

[1455] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches.

[1456] Processing user requests:

[1457] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server then deletes the relevant data from the database and updates the settings.

[1458] Specific example:

[1459] The user requests to delete an episode. The device sends this request to the server, which deletes the episode from the database. A notification is sent to the device upon completion of the deletion, and the user is shown "Episode deleted."

[1460] In this way, the system provides users with an interactive experience with their deceased loved ones, creating opportunities to share memories. Furthermore, by utilizing an emotion engine, it can respond in accordance with the user's emotions, resulting in a more personalized experience. With consideration for privacy and security, it provides a system that can be used with peace of mind.

[1461] The following describes the processing flow.

[1462] Creating a digital profile

[1463] Step 1:

[1464] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.).

[1465] Step 2:

[1466] The user completes the input and clicks the save button.

[1467] Step 3:

[1468] The terminal sends the entered information to the server in JSON format.

[1469] Step 4:

[1470] The server analyzes the received information and converts it to an appropriate format, such as Unicode.

[1471] Step 5:

[1472] The server saves the information to the database. After saving is complete, a notification is sent to the device confirming the save is finished.

[1473] Step 6:

[1474] The device receives a notification that saving is complete and displays "Digital profile saved" to the user.

[1475] AI reproduction

[1476] Step 1:

[1477] The user enters a message to start a conversation using the device's interface.

[1478] Step 2:

[1479] The user clicks the submit button.

[1480] Step 3:

[1481] The terminal sends the entered message to the server in JSON format.

[1482] Step 4:

[1483] The server receives the message and inputs it into the AI ​​model.

[1484] Step 5:

[1485] The server's AI model analyzes messages and generates appropriate responses. These responses include natural language processing and speech synthesis.

[1486] Step 6:

[1487] The server sends the generated response to the terminal in JSON format.

[1488] Step 7:

[1489] The device displays or plays the received response to the user via audio.

[1490] Memory Share

[1491] Step 1:

[1492] Users enter text, images, and videos into fields on their device interface to post episodes.

[1493] Step 2:

[1494] The user clicks the submit button.

[1495] Step 3:

[1496] The device sends the posted data to the server in JSON format.

[1497] Step 4:

[1498] The server analyzes the received episodes, converts them to an appropriate format, and stores them in the database.

[1499] Step 5:

[1500] The user enters a question about a specific memory.

[1501] Step 6:

[1502] The server searches the database for relevant episodes and retrieves the corresponding data.

[1503] Step 7:

[1504] The server's AI model generates natural-sounding responses based on the acquired episodes.

[1505] Step 8:

[1506] The server sends the generated response to the terminal in JSON format.

[1507] Step 9:

[1508] The device displays or plays aloud the response it receives to the user.

[1509] Emotion recognition by an emotion engine

[1510] Step 1:

[1511] When a user enters a text message, an engine works to analyze the emotions behind that message.

[1512] Step 2:

[1513] The terminal sends the entered message to the server in JSON format.

[1514] Step 3:

[1515] The server receives the message, and the emotion engine analyzes the message's content to recognize the emotional state (e.g., joy, sadness, anger).

[1516] Step 4:

[1517] Based on the emotions the server recognizes, it adjusts the AI ​​model, changing the tone and content of its responses.

[1518] Step 5:

[1519] The server sends the generated response to the terminal in JSON format.

[1520] Step 6:

[1521] The device displays or plays the received response to the user via audio.

[1522] Privacy and Security

[1523] Step 1:

[1524] The server encrypts all incoming information and stores it in the database.

[1525] Step 2:

[1526] Users can request data deletion or changes to their privacy settings.

[1527] Step 3:

[1528] The device sends the user's request to the server in JSON format.

[1529] Step 4:

[1530] The server searches the database for the relevant data and deletes it or updates the privacy settings upon the user's request.

[1531] Step 5:

[1532] The server notifies the terminal in JSON format that the request processing is complete.

[1533] Step 6:

[1534] Displays notifications received by the device to the user.

[1535] (Example 2)

[1536] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1537] In modern society, there is a need for systems that effectively preserve memories of the deceased and allow users to communicate with them interactively. Natural conversations based on the deceased's emotions and past events are particularly important, while ensuring data privacy and security remains a crucial challenge. Furthermore, technologies that can quickly and accurately analyze user input and generate appropriate responses are essential.

[1538] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1539] In this invention, the server includes means for analyzing messages sent by the user and generating responses using a generative AI model, means for analyzing the emotional state of the user's input message and adjusting the response based on the emotional state, and means for encrypting all received information and storing it in a database. This enables users to communicate with deceased loved ones in an interactive manner, share personal memories and episodes, and use the system with peace of mind.

[1540] A "user" is an individual who uses the system to input information about the deceased or to engage in conversations with them.

[1541] A "deceased person" is someone whose profile and memories are saved in the system, and who has passed away and who can be the subject of virtual conversations with the user.

[1542] An "interface" refers to the user interface or tools that a user uses to input information about a deceased person.

[1543] A "server" is a central computer system that receives information sent by users, processes it, and stores it in a database.

[1544] A "database" is a computer system used to structurally store and manage data such as information and anecdotes about a deceased person.

[1545] A "digital profile" is a collection of data, including the deceased person's name, age, photograph, audio sample, and personal anecdotes, which is stored in a database.

[1546] A "generative AI model" is an artificial intelligence algorithm used to analyze user messages and generate natural-sounding responses.

[1547] "Emotional state" refers to the type and intensity of emotions analyzed from the user's input messages.

[1548] "Encryption" is a technology that transforms data using specific algorithms to protect it from unauthorized access and leakage.

[1549] An "episode" is a detailed description or text of memories or events related to the deceased that the user has entered.

[1550] A "response" is a response message generated by a generative AI model and sent to the user.

[1551] "Privacy settings" are settings that allow users to define the scope of their data's visibility and use within the system.

[1552] This invention relates to a system that allows users to input information about a deceased person and engage in interactive conversations using a generated AI model based on that information. This system mainly consists of a terminal, a server, and a database.

[1553] Creating a digital profile

[1554] Terminal operation:

[1555] The user uses the device's interface to enter basic information about the deceased (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities. For example, the user might enter the deceased's name, "Mr. / Ms. A," a photo, a voice message, and their favorite events.

[1556] Server processing:

[1557] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. For example, the server receives a photo and voice message from "Person A" and stores them in the database to generate Person A's digital profile.

[1558] AI reproduction

[1559] Terminal operation:

[1560] The user enters a message to start a conversation on the terminal's interface. For example, they might type "Hello, A" and click the send button.

[1561] Server processing:

[1562] The server receives a message sent by the terminal. Next, the server analyzes the received message using a generative AI model (e.g., GPT-4) and generates a natural response based on the digital profile. The generated response is sent to the terminal and displayed to the user. For example, if the server receives the message "Hello, A," the generative AI model generates the response "Hello! How are you?" and sends it to the terminal.

[1563] Memory Share

[1564] Terminal operation:

[1565] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the post button. For example, a user might enter text and photos on the theme of "Last year's fireworks display."

[1566] Server processing:

[1567] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using a generative AI model based on the relevant information. For example, upon receiving the message, "Do you remember last year's fireworks display?", the server generates the response, "Of course, the fireworks that day were really beautiful."

[1568] Emotion recognition by an emotion engine

[1569] Terminal operation:

[1570] When a user types a text message, an engine works to analyze the emotion behind that message. For example, the user might type, "I'm really sad today."

[1571] Server processing:

[1572] When receiving a message sent from a device, the emotion engine analyzes the message content and recognizes the emotional state (e.g., joy, sadness, anger). Based on the recognized emotion, the server adjusts the generative AI model to change the tone and content of the response. For example, in response to a message that says, "I'm really sad today," the server might generate a comforting response such as, "It's okay, I'll listen."

[1573] Privacy and Security

[1574] Data encryption:

[1575] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches. For example, all transmitted data is encrypted using the TLS / SSL protocol.

[1576] Processing user requests:

[1577] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server deletes the relevant data from the database and updates the settings. For example, if a user requests to delete a specific episode, the server deletes that episode from the database and sends a message to the device stating, "Episode deleted."

[1578] This system allows users to enjoy interactive experiences with deceased loved ones and share personal memories. Furthermore, by utilizing an emotion engine and generative AI models, it provides personalized responses tailored to the user's emotions. In addition, the system is designed with privacy and security in mind, ensuring safe use.

[1579] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1580] Creating a digital profile

[1581] Terminal operation:

[1582] Step 1:

[1583] The user opens the device's interface. The user enters the deceased's name and age in the text fields on the interface, adds a photo using the file upload function, and records a voice sample using the voice recording function. This becomes the input data.

[1584] Server processing:

[1585] Step 2:

[1586] Once the user completes the information entry and clicks the save button, the device sends the entered information (text, photos, audio) to the server in JSON format. This is the input process.

[1587] Step 3:

[1588] The server parses the received JSON data and extracts each field (name, age, photo, audio, etc.). This parsing process prepares the data structure for storage in the database. Next, the server saves the parsed data to the database in the appropriate format. Specifically, it saves the name and age as text data, and the photo and audio as binary data. This is the output.

[1589] AI reproduction

[1590] Terminal operation:

[1591] Step 1:

[1592] The user opens the chat interface on their device and enters a message such as "Hello, A" into the text field. This is the input data.

[1593] Step 2:

[1594] When the user clicks the send button, the terminal sends the entered message to the server in JSON format. This is the input.

[1595] Server processing:

[1596] Step 3:

[1597] The server analyzes the received message and compares it to a digital profile. Based on the comparison results, the server uses a generative AI model (e.g., GPT-4) to create a prompt for generating an appropriate response. This prompt might take the form of, for example, "The user said 'Hello, Mr. / Ms. A' to the deceased. Please respond as the deceased." This is data processing.

[1598] Step 4:

[1599] The generated prompt is used to run the generative AI model, which then produces a natural-sounding response (e.g., "Hello! How are you?"). This is the output.

[1600] Step 5:

[1601] The server sends the generated response to the terminal in JSON format. This is also output.

[1602] Terminal operation:

[1603] Step 6:

[1604] The terminal displays the received response in the chat interface for the user to see. It also plays it back as audio if necessary. This is the final output.

[1605] Memory Share

[1606] Terminal operation:

[1607] Step 1:

[1608] Users input past episodes and memories using text fields or file upload functions. For example, they might input an episode titled "Last year's fireworks display" along with a photo. This is the input data.

[1609] Step 2:

[1610] When a user clicks the "Submit" button, the device sends the entered episode data to the server in JSON format. This is the input.

[1611] Server processing:

[1612] Step 3:

[1613] The server analyzes the received episode data and stores it in the database. The analysis process extracts the episode text and photo data, which are then inserted into the appropriate records in the database. This is the output.

[1614] Step 4:

[1615] When a user sends the message "Do you remember last year's fireworks display?" to the server, the server searches its database for relevant episodes. Specifically, it searches using the keyword "fireworks display" and extracts the corresponding data. This is the input data search.

[1616] Step 5:

[1617] Based on the episode information extracted by the server, a generative AI model is used to generate a response such as, "Of course, the fireworks that day were really beautiful." This is data processing.

[1618] Step 6:

[1619] The server sends the generated response to the terminal, which then displays it to the user. This is the final output.

[1620] Emotion recognition by an emotion engine

[1621] Terminal operation:

[1622] Step 1:

[1623] The user types a text message and sends it saying, "I'm really sad today." This is the input data.

[1624] Step 2:

[1625] When the user clicks the send button, the device sends the message to the server in JSON format. This is the input.

[1626] Server processing:

[1627] Step 3:

[1628] The server analyzes the received message and uses an emotion engine to recognize the user's emotional state as "sad." This emotion analysis process is a data calculation.

[1629] Step 4:

[1630] After the emotional state is recognized, the server adjusts the generating AI model to produce a comforting response such as, "It's okay, I'll listen." This is the output.

[1631] Step 5:

[1632] The server sends the generated response to the terminal, which then displays it to the user. It also plays it back as audio if necessary. This is the final output.

[1633] Privacy and Security

[1634] Data encryption:

[1635] Step 1:

[1636] The server encrypts all incoming information. Specifically, it encrypts the information using the SSL / TLS protocol and stores it in the database. This is the process of protecting the input data.

[1637] Processing user requests:

[1638] Step 2:

[1639] The user requests data deletion or changes to privacy settings from their device. For example, they might request, "I want to delete a specific episode." This is the input data.

[1640] Step 3:

[1641] The terminal sends a request to the server. This is the input.

[1642] Server processing:

[1643] Step 4:

[1644] The server analyzes the received request and deletes the corresponding data from the database. Specifically, it searches for the relevant episode based on the episode ID included in the request and physically deletes it from the database. This is data processing and output.

[1645] Step 5:

[1646] After the deletion is complete, the server notifies the terminal of the result. A message "Episode deleted" is generated and sent to the terminal. This is the final output.

[1647] This system provides users with an interactive experience with their deceased loved ones, creating opportunities to share memories. By utilizing an emotion engine and generative AI models, it can respond to the user's emotions, resulting in a more personalized experience. Furthermore, privacy and security measures are in place, ensuring safe and secure use.

[1648] (Application Example 2)

[1649] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1650] Traditional interactive systems for sharing memories of deceased loved ones lacked the ability to generate responses that took into account the user's emotions and location. This made it difficult to provide information about the deceased and their past in a more natural and personalized way. Furthermore, ensuring the privacy and security of information was insufficient. As a result, users were unable to enjoy a high level of immersion and security.

[1651] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for providing an interface to allow the user to input information about the deceased; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a digital profile of the deceased based on the stored information; means for generating a response using a generation AI model based on a message sent by the user; means for acquiring information at a specific spot using the customer's location information and generating a response; and means for transmitting the generated response to the user and playing it back by display or audio. This makes it possible to provide natural interaction tailored to the user's individual circumstances and location, ensuring the privacy and security of information while allowing for a higher level of immersion.

[1652] 1. A "user" is a person who uses this system to input, transmit, and manage information about the deceased.

[1653] 2. "Interface" refers to the screens, fields, and means of operating them that are necessary for a user to input information about the deceased.

[1654] 3. A "server" is a central computer system that processes information received from users and stores it in a database.

[1655] 4. A "database" is a data storage system used by a server to store, search, and manage information it receives.

[1656] 5. A "digital profile" refers to virtual personal data that includes basic information and memories of the deceased.

[1657] 6. A "generative AI model" is an artificial intelligence algorithm that analyzes messages sent by users and generates appropriate responses.

[1658] 7. A "response" is the answer or response that a server generates in response to user input or actions.

[1659] 8. "Location information" refers to GPS data that indicates the user's current location.

[1660] 9. A "spot" is a location associated with specific information or a particular response.

[1661] 10. "Audio playback" refers to a method of delivering a message generated as a response to the user as audio.

[1662] 11. An "episode" is a specific story or memory shared by a user about a deceased person or past events.

[1663] 12. A "message" is a piece of text or a request that a user inputs through the interface and sends to the server.

[1664] 13. "Sentiment analysis" refers to the process of recognizing a user's emotional state from their messages and statements.

[1665] 14. "Personalization" means providing customized support tailored to each individual user.

[1666] 15. "Privacy settings" are settings that allow users to decide how their information is shared and how their data is handled.

[1667] 16. "Encryption" is a technique that converts data into a format that cannot be read in order to securely protect it.

[1668] 17. The "episode submission field" is an input field where users can enter past events or memories.

[1669] The system for carrying out this invention consists of the following hardware and software.

[1670] Hardware and software used

[1671] Hardware: Smartphones, smart glasses, head-mounted displays

[1672] Software: AWS (Amazon Web Services), MySQL, Python, TensorFlow, OpenCV

[1673] System Configuration

[1674] The system primarily consists of user terminals, servers, and databases. The operation and role of each major component are described in detail below.

[1675] User terminal

[1676] The user terminal uses devices such as smartphones, smart glasses, and head-mounted displays. The user terminal provides an interface for inputting information about the deceased and functions for transmitting that information. This allows users to easily input data and send it to the server.

[1677] server

[1678] The server processes information received from the user and generates a digital profile. It also uses a generative AI model to produce natural-sounding responses based on the user's input messages. The server stores the received information in a MySQL database and searches this data as needed to generate the best possible answers.

[1679] database

[1680] The database plays a role in securely storing information received from the server. The database contains encrypted data such as information about deceased individuals, user-submitted episodes, and past message histories.

[1681] Processing flow

[1682] The user first uses an interface to enter information about the deceased. This interface includes text fields, a file upload function, and a voice recording function. The information entered by the user is sent to the server, analyzed, and stored in a database. This generates a digital profile of the deceased.

[1683] When a user stands at a specific location, the server uses GPS information from their smartphone or smart glasses to retrieve specific information. Based on this, a generative AI model generates a relevant response and sends it to the user's device. The user's device then displays or plays this response as text or audio.

[1684] Emotion analysis and personalization

[1685] When a user enters a message, the sentiment analysis engine activates to recognize the user's emotions. Based on these emotions, the generative AI model generates a personalized response. For example, if a user enters "I'm really sad today," the server performs sentiment analysis and, depending on the user's emotional state, generates a comforting response such as "It's okay, I'll listen" and sends it to the user's device.

[1686] Privacy and Security

[1687] All data is end-to-end encrypted and stored in the database. If a user requests data deletion or changes to privacy settings, the server responds immediately, deleting the relevant data from the database or updating the settings. This ensures users can use the system with peace of mind.

[1688] Examples of specific cases and prompt statements

[1689] As a concrete example, if a profile of a former regular customer named "Ichiro Tanaka" is registered, and a customer at the store says using smart glasses, "I've found Ichiro's usual seat," the AI ​​will respond, "Really? Ichiro used to drink coffee here often."

[1690] Example of a prompt:

[1691] "Tell me about this place."

[1692] "Tell me about Christmas events from the past."

[1693] "Please share your memories from here."

[1694] In this way, the system provides users with a personalized, interactive experience, deepening their connection with deceased loved ones and past events. Furthermore, security measures ensure that users can use the system with peace of mind.

[1695] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1696] Step 1:

[1697] The user uses the terminal interface to enter basic information about the deceased (name, age, photo, audio sample, personal anecdotes, etc.). The entered information is sent from the user's terminal to the server. Specifically, the user enters information into a text field and clicks the "Send" button. The input is text or multimedia files (photo, audio), and the output is a data packet sent to the server.

[1698] Step 2:

[1699] The server parses the received information and stores it in a MySQL database. The server converts the input data into the appropriate format and inserts it into the database using SQL statements. The input is the user's data sent, and the output is a digital profile stored in the database.

[1700] Step 3:

[1701] When a user stands at a specific location, the server uses the device's GPS information to obtain their location. The device sends current coordinate data to the server, which then uses this data to search for relevant information. The input is the GPS data sent from the device, and the output is the relevant information returned by the server.

[1702] Step 4:

[1703] The server generates a response using a generative AI model based on relevant information. Specifically, the server searches for relevant episodes in the database and inputs them as prompts into the generative AI model. The server then analyzes the text generated by the model and produces an appropriate response. The input is relevant information based on location data, and the output is the generated response.

[1704] Step 5:

[1705] The server sends the generated response to the user's terminal, which then displays or plays it as text or audio. Specifically, the server sends the generated response to the terminal, and the terminal presents the received data to the user in an appropriate format. The input is the generated response, and the output is the text display or audio playback to the user.

[1706] Step 6:

[1707] When a user enters and sends an additional message, the server receives it and uses its sentiment analysis engine to recognize the emotion. Based on the analysis results, the server adjusts its AI model to generate a response appropriate to the emotion. The input is the user's message, and the output is the emotion-appropriate response.

[1708] Step 7:

[1709] The server encrypts all data and securely stores it in the database. The input is the data to be stored, and the output is the encrypted data. When a user requests data deletion or changes to privacy settings, the server reflects this in the database and updates the settings. Specifically, it parses the user's request and deletes or updates the corresponding data from the database. The input is the user's request, and the output is the updated database state.

[1710] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1711] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1712] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1713] [Fourth Embodiment]

[1714] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1715] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1716] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1717] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1718] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1719] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1720] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1721] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1722] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1723] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1724] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1725] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1726] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1727] To implement this invention, the system is configured as follows.

[1728] Creating a digital profile

[1729] Terminal operation:

[1730] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities.

[1731] Server processing:

[1732] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. This generates a digital profile of the deceased person.

[1733] Specific example:

[1734] Users enter a photo of the deceased "Taro Yamada," an audio message from him, and information about events he enjoyed. The server receives the submitted information and stores it in a database as Taro Yamada's digital profile.

[1735] AI reproduction

[1736] Terminal operation:

[1737] The user uses a text box on the device interface to enter a message to start a conversation. They enter the message and click the send button to begin the conversation.

[1738] Server processing:

[1739] The device sends a message, which is then received. The server then uses an AI model to analyze the received message and generate a natural response based on the user's digital profile. The generated response is sent to the device and displayed to the user.

[1740] Specific example:

[1741] The user types and sends the message "Hello, Taro." The server receives this message and, through its AI model, generates the response "Hello! How are you?" which is then sent to the device. The device then displays or plays this response aloud.

[1742] Memory Share

[1743] Terminal operation:

[1744] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the submit button.

[1745] Server processing:

[1746] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using an AI model based on the relevant information.

[1747] Specific example:

[1748] The user types, "Do you remember the day we went to the beach last summer?" The server searches the database for relevant episodes and generates a response saying, "Of course, the beach was really beautiful that day."

[1749] Privacy and security, ethical considerations

[1750] Data encryption:

[1751] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches.

[1752] Processing user requests:

[1753] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server then deletes the relevant data from its database and updates the settings.

[1754] Specific example:

[1755] The user requests to delete an episode. The device sends this request to the server, which deletes the episode from the database. A notification is sent to the device upon completion of the deletion, and the user is shown "Episode deleted."

[1756] In this way, the system provides users with an interactive experience with their deceased loved ones, creating opportunities to share memories. It also prioritizes privacy and security, ensuring a system that can be used with peace of mind.

[1757] The following describes the processing flow.

[1758] Creating a digital profile

[1759] Step 1:

[1760] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.).

[1761] Step 2:

[1762] The user completes the input and clicks the save button.

[1763] Step 3:

[1764] The terminal sends the entered information to the server in JSON format.

[1765] Step 4:

[1766] The server analyzes the received information and converts it to an appropriate format, such as Unicode.

[1767] Step 5:

[1768] The server saves the information to the database. After saving is complete, a notification is sent to the device confirming the save is finished.

[1769] Step 6:

[1770] The device receives a notification that saving is complete and displays "Digital profile saved" to the user.

[1771] AI reproduction

[1772] Step 1:

[1773] The user enters a message to start a conversation using the device's interface.

[1774] Step 2:

[1775] The user clicks the submit button.

[1776] Step 3:

[1777] The terminal sends the entered message to the server in JSON format.

[1778] Step 4:

[1779] The server receives the message and inputs it into the AI ​​model.

[1780] Step 5:

[1781] The server's AI model analyzes messages and generates appropriate responses. These responses include natural language processing and speech synthesis.

[1782] Step 6:

[1783] The server sends the generated response to the terminal in JSON format.

[1784] Step 7:

[1785] The device displays or plays the received response to the user.

[1786] Memory Share

[1787] Step 1:

[1788] Users enter text, images, and videos into fields on their device interface to post episodes.

[1789] Step 2:

[1790] The user clicks the submit button.

[1791] Step 3:

[1792] The device sends the posted data to the server in JSON format.

[1793] Step 4:

[1794] The server analyzes the received episodes, converts them to an appropriate format, and stores them in the database.

[1795] Step 5:

[1796] The user enters a question about a specific memory.

[1797] Step 6:

[1798] The server searches the database for relevant episodes and retrieves the corresponding data.

[1799] Step 7:

[1800] The server's AI model generates natural-sounding responses based on the acquired episodes.

[1801] Step 8:

[1802] The server sends the generated response to the terminal in JSON format.

[1803] Step 9:

[1804] The device displays or plays the received response to the user.

[1805] Privacy and security, ethical considerations

[1806] Step 1:

[1807] The server encrypts all incoming information and stores it in the database.

[1808] Step 2:

[1809] Users can request data deletion or changes to their privacy settings.

[1810] Step 3:

[1811] The device sends the user's request to the server in JSON format.

[1812] Step 4:

[1813] The server searches the database for the relevant data and deletes it or updates the privacy settings upon the user's request.

[1814] Step 5:

[1815] The server notifies the terminal in JSON format that the request processing is complete.

[1816] Step 6:

[1817] The device displays the received notification to the user.

[1818] (Example 1)

[1819] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1820] Currently, many users are seeking ways to digitally manage and share memories and information about deceased loved ones, but this is difficult with conventional systems. Furthermore, many systems lack sufficient consideration for privacy and security, making them unsafe to use. Additionally, there is a lack of technology to generate natural conversations based on user input.

[1821] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1822] In this invention, the server includes means for providing an interface to allow the user to input information about a deceased person; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a digital profile of the deceased person based on the stored information; means for generating a response using a generative AI model based on a message sent by the user; means for sending the generated response to the user; means for the server to store the received information using AES-256 encryption technology; means for deleting the relevant information from the database based on a deletion request from the user; and means for updating the user's privacy settings. As a result, the user can safely manage and share information about a deceased person and also enjoy natural conversations through the generative AI model.

[1823] A "user" refers to an individual who uses this system to input, manage, and share information about a deceased person.

[1824] "Interface" refers to the screens and operating methods that users use to access a system and input and manage information.

[1825] A "server" refers to a computer system that analyzes received information, stores it in a database, and generates responses using generative AI models.

[1826] A "database" refers to a digital information management system used to store and manage information about deceased individuals and episodes posted by users.

[1827] A "digital profile" refers to a collection of data that compiles information such as the name, age, photograph, audio sample, and personal anecdotes of a deceased person.

[1828] A "generative AI model" refers to an artificial intelligence model used to analyze user input messages and generate natural-sounding responses.

[1829] "AES-256 encryption technology" refers to a strong encryption technology used to keep received information secure.

[1830] A "deletion request" refers to the operation in which a user sends a request to a server to delete their own data.

[1831] "Privacy settings" refer to the options and actions that allow users to set the scope of visibility and access rights for their data.

[1832] To implement this invention, the system is configured as follows: First, the user inputs basic information about the deceased person using the terminal interface. This interface includes a text field, a file upload function, a voice recording function, and the like.

[1833] Creating a digital profile

[1834] Terminal operation:

[1835] The user opens the terminal interface and enters the deceased person's name, age, photo, audio sample, and personal anecdotes. Once the input is complete, they click the save button. At this point, the terminal converts the entered information into JSON format and sends it to the server as an HTTP request.

[1836] Server processing:

[1837] The server receives an HTTP request, parses the received content, and separates the data into individual fields. Photos are saved as binary data, and audio is saved as an audio file. The analyzed data is stored in a database using an RDBMS such as MySQL. This generates a digital profile of the deceased person.

[1838] Specific example:

[1839] The user enters information about "Deceased Person A" and uploads a photo and an audio message. When the user clicks the save button, the device sends this information to the server. The server analyzes the information and saves it to a database, completing the digital profile of "Deceased Person A".

[1840] AI reproduction

[1841] Terminal operation:

[1842] The user opens the terminal interface and types a message to start a conversation into the text box. After typing the message, they click the send button.

[1843] Server processing:

[1844] The terminal converts the message it sends into JSON format and sends it to the server. The server analyzes and processes the received message using a generative AI model (e.g., GPT-3 or ChatGPT). The model generates a natural response, which the server converts back into JSON format and sends back to the terminal. The terminal displays the received response and plays it back as audio if necessary.

[1845] Specific example:

[1846] The user types and sends the message "Hello, Deceased Person A". The server receives this message and uses a generative AI model to generate the response "Hello! How are you?" and sends it to the device. The device displays and plays this response aloud.

[1847] Memory Share

[1848] Terminal operation:

[1849] Users use text fields and file upload functions to enter past memories and episodes. They enter the necessary information and click the submit button.

[1850] Server processing:

[1851] When the device recognizes a click of the post button, it converts the episode information into JSON format and sends it to the server as an HTTP request. The server stores the submitted episode in its database. If the user later initiates a conversation about a particular episode, the server searches and extracts the relevant episode from the database and generates a response using a generative AI model.

[1852] Specific example:

[1853] The user types, "Do you remember the day we went to the beach last summer?" The server searches the database for relevant episodes and generates a response saying, "Of course, the beach was really beautiful that day."

[1854] Privacy and security, ethical considerations

[1855] Data encryption:

[1856] The server stores all received information in a database using AES-256 encryption technology. This prevents unauthorized access and data breaches.

[1857] Processing user requests:

[1858] When a user requests data deletion or changes to privacy settings, the device converts this data into JSON format and sends it to the server as an HTTP request. The server receives the request, deletes the data from the database, and updates the settings. If the deletion or update is successful, the server sends a notification back to the device, and the user is shown a message confirming the completion of the deletion.

[1859] Specific example:

[1860] The user requests to delete an episode. The device sends this request to the server, which then deletes the data from the database. A notification is sent to the device upon completion of the deletion, displaying the message "Episode deleted."

[1861] Example of a prompt

[1862] Creating a digital profile: The deceased person's name, age, photo, voice sample, personal anecdotes, etc., are entered on a device and saved to the server's database.

[1863] AI-powered reproduction: The user enters a message, and the server generates a natural response using an AI model.

[1864] Memory sharing: Users post past episodes from their devices, and the server saves them to a database.

[1865] Privacy and Security: The server encrypts all incoming information and processes user data deletion requests.

[1866] In this way, the system provides users with an interactive experience with deceased loved ones, creating opportunities to share memories, while also ensuring a safe and secure system that respects privacy and security.

[1867] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1868] Creating a digital profile

[1869] Step 1:

[1870] The user opens the device's interface and enters the deceased person's name, age, photo, voice sample, personal anecdotes, etc. The entered information is temporarily stored in the device's local memory.

[1871] Input: User enters name, age, photo, voice sample, and episode.

[1872] Output: Information temporarily stored in the terminal's local memory.

[1873] Step 2:

[1874] After the user has entered all the information, they click the save button. The device converts the entered information into JSON format and sends it to the server as an HTTP request.

[1875] Input: Click the save button.

[1876] Output: Information in JSON format sent to the server.

[1877] Step 3:

[1878] The server receives an HTTP request, parses the received content, and divides the data into individual fields. For example, it might save the name as a string field, the photo as binary data, and the audio as an audio file.

[1879] Input: Information sent to the server in JSON format.

[1880] Output: Data for each field analyzed within the server.

[1881] Step 4:

[1882] The server saves the analyzed data to a database using an RDBMS such as MySQL. If the save is successful, the server sends a success message to the terminal in JSON format.

[1883] Input: Data separated into each field.

[1884] Output: Data stored in the database and a success message sent to the terminal.

[1885] AI reproduction

[1886] Step 1:

[1887] The user opens the terminal interface and types a message to start a conversation into the text box. After typing the message, they click the send button.

[1888] Input: User input of a text message.

[1889] Output: Messages temporarily stored within the device.

[1890] Step 2:

[1891] The device recognizes the click of the send button, converts the message to JSON format, and sends it to the server as an HTTP request.

[1892] Input: Click the submit button.

[1893] Output: A JSON-formatted message sent to the server.

[1894] Step 3:

[1895] The server analyzes and processes received messages using a generating AI model (e.g., GPT-3 or ChatGPT). The model then generates a natural-sounding response.

[1896] Input: A JSON-formatted message sent to the server.

[1897] Output: Response generated through the generative AI model.

[1898] Step 4:

[1899] The server converts the generated response into JSON format and sends it to the terminal as an HTTP response. The terminal displays the received response and plays it aloud if necessary.

[1900] Input: The generated response.

[1901] Output: A JSON-formatted response sent to the terminal, along with its display and audio playback on the terminal.

[1902] Memory Share

[1903] Step 1:

[1904] The user opens the terminal interface and uses text fields and file upload functions to enter past memories and episodes. They enter the necessary information and click the submit button.

[1905] Input: User input of episodes and files.

[1906] Output: Episode information temporarily stored on the device.

[1907] Step 2:

[1908] The device recognizes the click of the post button, converts the episode information into JSON format, and sends it to the server as an HTTP request.

[1909] Input: Click the submit button.

[1910] Output: Episode information in JSON format sent to the server.

[1911] Step 3:

[1912] The server saves the received episode to the database. After saving, the server sends a success message to the terminal in JSON format.

[1913] Input: Episode information in JSON format sent to the server.

[1914] Output: Episode information stored in the database and a success message sent to the terminal.

[1915] Step 4:

[1916] If a user later initiates a conversation based on a specific episode, the server searches and extracts the relevant episode from the database and generates a response using a generative AI model.

[1917] Input: User-submitted messages based on their experiences.

[1918] Output: Natural responses generated through a generative AI model.

[1919] Privacy and security, ethical considerations

[1920] Step 1:

[1921] The server stores all received information in a database using AES-256 encryption technology. This prevents unauthorized access and data breaches.

[1922] Input: Information sent to the server.

[1923] Output: Database storage encrypted using AES-256 technology.

[1924] Step 2:

[1925] When a user requests data deletion or changes to privacy settings, the device converts this information into JSON format and sends it to the server as an HTTP request.

[1926] Input: User request for deletion or change of privacy settings.

[1927] Output: A JSON-formatted request sent to the server.

[1928] Step 3:

[1929] The server receives the request, deletes the relevant data from the database, and updates the settings. If the deletion or update is successful, the server sends a success message to the terminal.

[1930] Input: Deletion or privacy setting change request in JSON format.

[1931] Output: Data deleted from the database, or updated privacy settings, and a success message notified to the user.

[1932] (Application Example 1)

[1933] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1934] Currently, many users wish to share memories of deceased family and friends and maintain emotional connections through dialogue. However, existing systems lack the technology to meticulously profile information about deceased individuals and generate natural conversations using AI. Furthermore, concerns about privacy and security necessitate a safe and secure system. This invention aims to solve these problems and provide a system that allows users to engage in natural virtual dialogues with deceased individuals.

[1935] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1936] In this invention, the server includes means for providing an interface to allow the user to input information about a deceased person; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a profile of the deceased person based on the stored information; means for generating a response using a generative AI model based on a message sent by the user; means for sending the generated response to the user; means for encrypting and decrypting the profile data; means for storing past episodes and having the generative AI model generate responses based on these; and means for generating and sending prompt messages to the user's terminal. This enables the user to have a natural conversation with the deceased person and share memories. Furthermore, it enables safe use that takes privacy and security into consideration.

[1937] A "user" is someone who uses the system to input information about a deceased person and then engages in subsequent conversations and shares stories about them.

[1938] An "interface" refers to the screen or control panel that a user uses to operate a device, input information, and transmit it.

[1939] A "profile" is a collection of information that includes the name, age, photograph, audio sample, and personal anecdotes of a deceased person.

[1940] A "generative AI model" is an artificial intelligence model used to generate natural-sounding responses based on user input messages and referencing the profiles of deceased individuals.

[1941] A "response" refers to the reply or reaction from a generative AI model to a message sent by a user.

[1942] A "database" is an electronic information management system used by a server to store information and episodes it receives.

[1943] "Encryption" is the process of transforming data using special techniques to make it unreadable, in order to protect it from unauthorized access.

[1944] "Decryption" is the process of restoring encrypted data to its original, readable state.

[1945] An "episode" is a user-submitted memory or event related to a deceased person.

[1946] A "prompt sentence" is an input sentence for a generative AI model, and it is a sentence necessary to generate an accurate response based on the user's message.

[1947] A "terminal" refers to a user-operated device such as a smartphone, and is used to provide the interface for this system.

[1948] The system necessary to implement this invention is configured as follows: The system mainly consists of a user terminal, a server, a database, and a generative AI model.

[1949] Creating a digital profile

[1950] The user first uses the terminal's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). Specifically, they use the text fields, file upload function, and voice recording function on the terminal's screen. This information is sent to the server when the user clicks the save button. The server analyzes the received information and stores it in a database, generating a digital profile of the deceased person.

[1951] AI reproduction

[1952] The user enters a message to initiate a conversation through the terminal's interface. The server receives the message sent from the terminal and analyzes it using a generative AI model. It then generates a natural response based on the user's digital profile and sends it to the terminal for display to the user.

[1953] Memory Share

[1954] Users use text fields and file upload functions to post past memories and episodes. By entering the necessary information and clicking the submit button, the information is sent to the server. The server stores the received episodes in a database. Subsequently, when a user starts a conversation based on a particular episode, the server searches for previously stored episodes and generates a response using a generative AI model based on the relevant information.

[1955] Privacy and security, ethical considerations

[1956] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data leaks. Furthermore, if a user requests data deletion or changes to privacy settings, the device sends this request to the server, which then deletes the relevant data from the database and updates the settings.

[1957] To give a specific example, when a user types and sends the message "Hello, Taro," the server receives this message, generates a response "Hello! How are you?" through a generative AI model, and sends and displays it on the device.

[1958] Furthermore, if you want to talk about past memories, you can type something like, "Do you remember the day we went to the beach last summer?" The server will search its database for relevant episodes and generate a response like, "Of course, the beach was really beautiful that day."

[1959] Specific examples of hardware and software include smartphones (iOS, Android) as terminals, Flask as a server-side framework, SQLite and PostgreSQL as databases, and GPT-3 (using the Hugging Face transformers library) as a generative AI model.

[1960] Specific examples of prompt messages include user messages such as "What are you doing today?". Questions like "How are you doing?" and messages that evoke past events, such as "Do you remember the day we went to the beach last summer?" are also used as prompt messages.

[1961] This allows users to have natural conversations with deceased loved ones and share memories. Furthermore, it enables safe use with consideration for privacy and security.

[1962] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1963] Step 1:

[1964] The user operates the device's interface to enter basic information about the deceased person. This interface includes text fields, a file upload function, and a voice recording function. Once all the information is entered, the user clicks the save button to confirm the input. Examples of input data include name, age, photo, voice sample, and personal anecdotes.

[1965] Step 2:

[1966] The terminal sends information entered by the user to the server. During this process, the terminal converts the input data into JSON format and transmits it securely using the HTTPS protocol. The data received by the server includes name, age, photo, voice sample, and episode.

[1967] Step 3:

[1968] The server parses the received information and stores it in a database. The server encrypts the data using encryption techniques and later stores it in an appropriate format in each field to maintain data integrity. This data is stored in an SQL database (e.g., PostgreSQL). The input here is user data, and the output is encrypted data stored in the database.

[1969] Step 4:

[1970] The user enters a message to initiate a conversation on the terminal interface. By typing the message in the text box and clicking the send button, the message is sent to the server. Examples of specific prompt messages include "Hello, how are you?".

[1971] Step 5:

[1972] The server analyzes messages received from the terminal. First, it decodes and retrieves the target profile from the database and formats it into an appropriate format for input into a generating AI model (e.g., GPT-3). The input data consists of messages from the user and profile information, and the output is a prompt message provided to the generating AI model.

[1973] Step 6:

[1974] The server uses a generative AI model to generate a natural response from the prompt. The generative AI model generates an appropriate response based on profile information and the user's message. This process results in a response such as, "Hello! How are you?"

[1975] Step 7:

[1976] The server sends the generated response to the user's terminal. The response is encoded in JSON format and sent securely using the HTTPS protocol. The input data is the generated response message, and the output data is the response displayed on the user's terminal.

[1977] Step 8:

[1978] The device displays the response received by the user. This can be done by displaying the message as text in the UI or playing it as audio. This allows the user to continue the conversation.

[1979] Step 9:

[1980] The system uses text fields and file upload functions to allow users to post past memories and episodes. Users confirm their input data with a submit button, and the device sends the data to the server. The data sent includes memories and episodes.

[1981] Step 10:

[1982] The server saves the received episodes to the database. The data is encrypted during this process. The input data is the episode information, and the output data is the encrypted episode information stored in the database.

[1983] Step 11:

[1984] When a user initiates a conversation based on a past episode, the server searches the database for the relevant episode and generates a corresponding response using a generative AI model. For example, if a user types, "Do you remember the day we went to the beach last summer?", the server searches for the relevant episode and generates a response such as, "Of course, the beach was really beautiful that day."

[1985] Step 12:

[1986] The server processes data encryption, decryption, and deletion requests based on the user's privacy and security settings. This prevents unauthorized access and data breaches, ensuring secure use.

[1987] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1988] To carry out this invention, the system consists of the following elements.

[1989] Creating a digital profile

[1990] Terminal operation:

[1991] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities.

[1992] Server processing:

[1993] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. This generates a digital profile of the deceased person.

[1994] Specific example:

[1995] Users input a photo of the deceased "Ichiro Tanaka," along with audio messages from his lifetime and information about events he enjoyed. The server receives the submitted information and stores it in a database as Ichiro Tanaka's digital profile.

[1996] AI reproduction

[1997] Terminal operation:

[1998] The user types a message to start a conversation on the device's interface. The conversation begins when the user clicks the send button.

[1999] Server processing:

[2000] The device sends a message, which is then received. The server then uses an AI model to analyze the received message and generate a natural response based on the user's digital profile. The generated response is sent to the device and displayed to the user.

[2001] Specific example:

[2002] The user types and sends the message "Hello, Ichiro". The server receives this message and, through its AI model, generates the response "Hello! How are you?" and sends it to the device. The device then displays or plays this response aloud.

[2003] Memory Share

[2004] Terminal operation:

[2005] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the submit button.

[2006] Server processing:

[2007] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using an AI model based on the relevant information.

[2008] Specific example:

[2009] The user types, "Do you remember last year's fireworks display?" The server searches the database for relevant episodes and generates a response saying, "Of course, the fireworks that day were really beautiful."

[2010] Emotion recognition by an emotion engine

[2011] Terminal operation:

[2012] When a user enters a text message, an engine works to analyze the emotions behind that message.

[2013] Server processing:

[2014] When receiving a message sent from a device, the emotion engine analyzes the message content and recognizes the emotional state (e.g., joy, sadness, anger). Based on the recognized emotion, the server adjusts the AI ​​model, changing the tone and content of the response.

[2015] Specific example:

[2016] The user sends the message, "I'm really sad today." The server's emotion engine analyzes this message and recognizes that the user is in a state of sadness. Based on this information, the server generates a comforting response, "It's okay, I'll listen," and sends it to the device.

[2017] Privacy and Security

[2018] Data encryption:

[2019] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches.

[2020] Processing user requests:

[2021] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server then deletes the relevant data from the database and updates the settings.

[2022] Specific example:

[2023] The user requests to delete an episode. The device sends this request to the server, which deletes the episode from the database. A notification is sent to the device upon completion of the deletion, and the user is shown "Episode deleted."

[2024] In this way, the system provides users with an interactive experience with their deceased loved ones, creating opportunities to share memories. Furthermore, by utilizing an emotion engine, it can respond in accordance with the user's emotions, resulting in a more personalized experience. With consideration for privacy and security, it provides a system that can be used with peace of mind.

[2025] The following describes the processing flow.

[2026] Creating a digital profile

[2027] Step 1:

[2028] The user uses the device's interface to enter basic information about the deceased person (name, age, photo, voice sample, personal anecdotes, etc.).

[2029] Step 2:

[2030] The user completes the input and clicks the save button.

[2031] Step 3:

[2032] The terminal sends the entered information to the server in JSON format.

[2033] Step 4:

[2034] The server analyzes the received information and converts it to an appropriate format, such as Unicode.

[2035] Step 5:

[2036] The server saves the information to the database. After saving is complete, a notification is sent to the device confirming the save is finished.

[2037] Step 6:

[2038] The device receives a notification that saving is complete and displays "Digital profile saved" to the user.

[2039] AI reproduction

[2040] Step 1:

[2041] The user enters a message to start a conversation using the device's interface.

[2042] Step 2:

[2043] The user clicks the submit button.

[2044] Step 3:

[2045] The terminal sends the entered message to the server in JSON format.

[2046] Step 4:

[2047] The server receives the message and inputs it into the AI ​​model.

[2048] Step 5:

[2049] The server's AI model analyzes messages and generates appropriate responses. These responses include natural language processing and speech synthesis.

[2050] Step 6:

[2051] The server sends the generated response to the terminal in JSON format.

[2052] Step 7:

[2053] The device displays or plays the received response to the user via audio.

[2054] Memory Share

[2055] Step 1:

[2056] Users enter text, images, and videos into fields on their device interface to post episodes.

[2057] Step 2:

[2058] The user clicks the submit button.

[2059] Step 3:

[2060] The device sends the posted data to the server in JSON format.

[2061] Step 4:

[2062] The server analyzes the received episodes, converts them to an appropriate format, and stores them in the database.

[2063] Step 5:

[2064] The user enters a question about a specific memory.

[2065] Step 6:

[2066] The server searches the database for relevant episodes and retrieves the corresponding data.

[2067] Step 7:

[2068] The server's AI model generates natural-sounding responses based on the acquired episodes.

[2069] Step 8:

[2070] The server sends the generated response to the terminal in JSON format.

[2071] Step 9:

[2072] The device displays or plays aloud the response it receives to the user.

[2073] Emotion recognition by an emotion engine

[2074] Step 1:

[2075] When a user enters a text message, an engine works to analyze the emotions behind that message.

[2076] Step 2:

[2077] The terminal sends the entered message to the server in JSON format.

[2078] Step 3:

[2079] The server receives the message, and the emotion engine analyzes the message's content to recognize the emotional state (e.g., joy, sadness, anger).

[2080] Step 4:

[2081] Based on the emotions the server recognizes, it adjusts the AI ​​model, changing the tone and content of its responses.

[2082] Step 5:

[2083] The server sends the generated response to the terminal in JSON format.

[2084] Step 6:

[2085] The device displays or plays the received response to the user via audio.

[2086] Privacy and Security

[2087] Step 1:

[2088] The server encrypts all incoming information and stores it in the database.

[2089] Step 2:

[2090] Users can request data deletion or changes to their privacy settings.

[2091] Step 3:

[2092] The device sends the user's request to the server in JSON format.

[2093] Step 4:

[2094] The server searches the database for the relevant data and deletes it or updates the privacy settings upon the user's request.

[2095] Step 5:

[2096] The server notifies the terminal in JSON format that the request processing is complete.

[2097] Step 6:

[2098] Displays notifications received by the device to the user.

[2099] (Example 2)

[2100] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2101] In modern society, there is a need for systems that effectively preserve memories of the deceased and allow users to communicate with them interactively. Natural conversations based on the deceased's emotions and past events are particularly important, while ensuring data privacy and security remains a crucial challenge. Furthermore, technologies that can quickly and accurately analyze user input and generate appropriate responses are essential.

[2102] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[2103] In this invention, the server includes means for analyzing messages sent by the user and generating responses using a generative AI model, means for analyzing the emotional state of the user's input message and adjusting the response based on the emotional state, and means for encrypting all received information and storing it in a database. This enables users to communicate with deceased loved ones in an interactive manner, share personal memories and episodes, and use the system with peace of mind.

[2104] A "user" is an individual who uses the system to input information about the deceased or to engage in conversations with them.

[2105] A "deceased person" is someone whose profile and memories are saved in the system, and who has passed away and who can be the subject of virtual conversations with the user.

[2106] An "interface" refers to the user interface or tools that a user uses to input information about a deceased person.

[2107] A "server" is a central computer system that receives information sent by users, processes it, and stores it in a database.

[2108] A "database" is a computer system used to structurally store and manage data such as information and anecdotes about a deceased person.

[2109] A "digital profile" is a collection of data, including the deceased person's name, age, photograph, audio sample, and personal anecdotes, which is stored in a database.

[2110] A "generative AI model" is an artificial intelligence algorithm used to analyze user messages and generate natural-sounding responses.

[2111] "Emotional state" refers to the type and intensity of emotions analyzed from the user's input messages.

[2112] "Encryption" is a technology that transforms data using specific algorithms to protect it from unauthorized access and leakage.

[2113] An "episode" is a detailed description or text of memories or events related to the deceased that the user has entered.

[2114] A "response" is a response message generated by a generative AI model and sent to the user.

[2115] "Privacy settings" are settings that allow users to define the scope of their data's visibility and use within the system.

[2116] This invention relates to a system that allows users to input information about a deceased person and engage in interactive conversations using a generated AI model based on that information. This system mainly consists of a terminal, a server, and a database.

[2117] Creating a digital profile

[2118] Terminal operation:

[2119] The user uses the device's interface to enter basic information about the deceased (name, age, photo, voice sample, personal anecdotes, etc.). This interface includes text fields, file upload functionality, and voice recording capabilities. For example, the user might enter the deceased's name, "Mr. / Ms. A," a photo, a voice message, and their favorite events.

[2120] Server processing:

[2121] Once the user completes the input and clicks the save button, the device sends the entered information to the server. The server analyzes the received information and stores it in a database. For example, the server receives a photo and voice message from "Person A" and stores them in the database to generate Person A's digital profile.

[2122] AI reproduction

[2123] Terminal operation:

[2124] The user enters a message to start a conversation on the terminal's interface. For example, they might type "Hello, A" and click the send button.

[2125] Server processing:

[2126] The server receives a message sent by the terminal. Next, the server analyzes the received message using a generative AI model (e.g., GPT-4) and generates a natural response based on the digital profile. The generated response is sent to the terminal and displayed to the user. For example, if the server receives the message "Hello, A," the generative AI model generates the response "Hello! How are you?" and sends it to the terminal.

[2127] Memory Share

[2128] Terminal operation:

[2129] Users use text fields and file upload functions to post past memories and episodes. They enter the necessary information and click the post button. For example, a user might enter text and photos on the theme of "Last year's fireworks display."

[2130] Server processing:

[2131] The system receives submitted episodes and memories and stores them in a database. Then, when a user initiates a conversation based on a specific episode, the server searches for previously stored episodes and generates a response using a generative AI model based on the relevant information. For example, upon receiving the message, "Do you remember last year's fireworks display?", the server generates the response, "Of course, the fireworks that day were really beautiful."

[2132] Emotion recognition by an emotion engine

[2133] Terminal operation:

[2134] When a user types a text message, an engine works to analyze the emotion behind that message. For example, the user might type, "I'm really sad today."

[2135] Server processing:

[2136] When receiving a message sent from a device, the emotion engine analyzes the message content and recognizes the emotional state (e.g., joy, sadness, anger). Based on the recognized emotion, the server adjusts the generative AI model to change the tone and content of the response. For example, in response to a message that says, "I'm really sad today," the server might generate a comforting response such as, "It's okay, I'll listen."

[2137] Privacy and Security

[2138] Data encryption:

[2139] The server encrypts all incoming information and stores it in a database. This prevents unauthorized access and data breaches. For example, all transmitted data is encrypted using the TLS / SSL protocol.

[2140] Processing user requests:

[2141] When a user requests data deletion or changes to privacy settings, the device sends this request to the server. The server deletes the relevant data from the database and updates the settings. For example, if a user requests to delete a specific episode, the server deletes that episode from the database and sends a message to the device stating, "Episode deleted."

[2142] This system allows users to enjoy interactive experiences with deceased loved ones and share personal memories. Furthermore, by utilizing an emotion engine and generative AI models, it provides personalized responses tailored to the user's emotions. In addition, the system is designed with privacy and security in mind, ensuring safe use.

[2143] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2144] Creating a digital profile

[2145] Terminal operation:

[2146] Step 1:

[2147] The user opens the device's interface. The user enters the deceased's name and age in the text fields on the interface, adds a photo using the file upload function, and records a voice sample using the voice recording function. This becomes the input data.

[2148] Server processing:

[2149] Step 2:

[2150] Once the user completes the information entry and clicks the save button, the device sends the entered information (text, photos, audio) to the server in JSON format. This is the input process.

[2151] Step 3:

[2152] The server parses the received JSON data and extracts each field (name, age, photo, audio, etc.). This parsing process prepares the data structure for storage in the database. Next, the server saves the parsed data to the database in the appropriate format. Specifically, it saves the name and age as text data, and the photo and audio as binary data. This is the output.

[2153] AI reproduction

[2154] Terminal operation:

[2155] Step 1:

[2156] The user opens the chat interface on their device and enters a message such as "Hello, A" into the text field. This is the input data.

[2157] Step 2:

[2158] When the user clicks the send button, the terminal sends the entered message to the server in JSON format. This is the input.

[2159] Server processing:

[2160] Step 3:

[2161] The server analyzes the received message and compares it to a digital profile. Based on the comparison results, the server uses a generative AI model (e.g., GPT-4) to create a prompt for generating an appropriate response. This prompt might take the form of, for example, "The user said 'Hello, Mr. / Ms. A' to the deceased. Please respond as the deceased." This is data processing.

[2162] Step 4:

[2163] The generated prompt is used to run the generative AI model, which then produces a natural-sounding response (e.g., "Hello! How are you?"). This is the output.

[2164] Step 5:

[2165] The server sends the generated response to the terminal in JSON format. This is also output.

[2166] Terminal operation:

[2167] Step 6:

[2168] The terminal displays the received response in the chat interface for the user to see. It also plays it back as audio if necessary. This is the final output.

[2169] Memory Share

[2170] Terminal operation:

[2171] Step 1:

[2172] Users input past episodes and memories using text fields or file upload functions. For example, they might input an episode titled "Last year's fireworks display" along with a photo. This is the input data.

[2173] Step 2:

[2174] When a user clicks the "Submit" button, the device sends the entered episode data to the server in JSON format. This is the input.

[2175] Server processing:

[2176] Step 3:

[2177] The server analyzes the received episode data and stores it in the database. The analysis process extracts the episode text and photo data, which are then inserted into the appropriate records in the database. This is the output.

[2178] Step 4:

[2179] When a user sends the message "Do you remember last year's fireworks display?" to the server, the server searches its database for relevant episodes. Specifically, it searches using the keyword "fireworks display" and extracts the corresponding data. This is the input data search.

[2180] Step 5:

[2181] Based on the episode information extracted by the server, a generative AI model is used to generate a response such as, "Of course, the fireworks that day were really beautiful." This is data processing.

[2182] Step 6:

[2183] The server sends the generated response to the terminal, which then displays it to the user. This is the final output.

[2184] Emotion recognition by an emotion engine

[2185] Terminal operation:

[2186] Step 1:

[2187] The user types a text message and sends it saying, "I'm really sad today." This is the input data.

[2188] Step 2:

[2189] When the user clicks the send button, the device sends the message to the server in JSON format. This is the input.

[2190] Server processing:

[2191] Step 3:

[2192] The server analyzes the received message and uses an emotion engine to recognize the user's emotional state as "sad." This emotion analysis process is a data calculation.

[2193] Step 4:

[2194] After the emotional state is recognized, the server adjusts the generating AI model to produce a comforting response such as, "It's okay, I'll listen." This is the output.

[2195] Step 5:

[2196] The server sends the generated response to the terminal, which then displays it to the user. It also plays it back as audio if necessary. This is the final output.

[2197] Privacy and Security

[2198] Data encryption:

[2199] Step 1:

[2200] The server encrypts all incoming information. Specifically, it encrypts the information using the SSL / TLS protocol and stores it in the database. This is the process of protecting the input data.

[2201] Processing user requests:

[2202] Step 2:

[2203] The user requests data deletion or changes to privacy settings from their device. For example, they might request, "I want to delete a specific episode." This is the input data.

[2204] Step 3:

[2205] The terminal sends a request to the server. This is the input.

[2206] Server processing:

[2207] Step 4:

[2208] The server analyzes the received request and deletes the corresponding data from the database. Specifically, it searches for the relevant episode based on the episode ID included in the request and physically deletes it from the database. This is data processing and output.

[2209] Step 5:

[2210] After the deletion is complete, the server notifies the terminal of the result. A message "Episode deleted" is generated and sent to the terminal. This is the final output.

[2211] This system provides users with an interactive experience with their deceased loved ones, creating opportunities to share memories. By utilizing an emotion engine and generative AI models, it can respond to the user's emotions, resulting in a more personalized experience. Furthermore, privacy and security measures are in place, ensuring safe and secure use.

[2212] (Application Example 2)

[2213] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2214] Traditional interactive systems for sharing memories of deceased loved ones lacked the ability to generate responses that took into account the user's emotions and location. This made it difficult to provide information about the deceased and their past in a more natural and personalized way. Furthermore, ensuring the privacy and security of information was insufficient. As a result, users were unable to enjoy a high level of immersion and security.

[2215] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for providing an interface to allow the user to input information about the deceased; means for transmitting the information entered by the user to the server; means for the server to store the received information in a database; means for generating a digital profile of the deceased based on the stored information; means for generating a response using a generation AI model based on a message sent by the user; means for acquiring information at a specific spot using the customer's location information and generating a response; and means for transmitting the generated response to the user and playing it back by display or audio. This makes it possible to provide natural interaction tailored to the user's individual circumstances and location, ensuring the privacy and security of information while allowing for a higher level of immersion.

[2216] 1. A "user" is a person who uses this system to input, transmit, and manage information about the deceased.

[2217] 2. "Interface" refers to the screens, fields, and means of operating them that are necessary for a user to input information about the deceased.

[2218] 3. A "server" is a central computer system that processes information received from users and stores it in a database.

[2219] 4. A "database" is a data storage system used by a server to store, search, and manage information it receives.

[2220] 5. A "digital profile" refers to virtual personal data that includes basic information and memories of the deceased.

[2221] 6. A "generative AI model" is an artificial intelligence algorithm that analyzes messages sent by users and generates appropriate responses.

[2222] 7. A "response" is the answer or response that a server generates in response to user input or actions.

[2223] 8. "Location information" refers to GPS data that indicates the user's current location.

[2224] 9. A "spot" is a location associated with specific information or a particular response.

[2225] 10. "Audio playback" refers to a method of delivering a message generated as a response to the user as audio.

[2226] 11. An "episode" is a specific story or memory shared by a user about a deceased person or past events.

[2227] 12. A "message" is a piece of text or a request that a user inputs through the interface and sends to the server.

[2228] 13. "Sentiment analysis" refers to the process of recognizing a user's emotional state from their messages and statements.

[2229] 14. "Personalization" means providing customized support tailored to each individual user.

[2230] 15. "Privacy settings" are settings that allow users to decide how their information is shared and how their data is handled.

[2231] 16. "Encryption" is a technique that converts data into a format that cannot be read in order to securely protect it.

[2232] 17. The "episode submission field" is an input field where users can enter past events or memories.

[2233] The system for carrying out this invention consists of the following hardware and software.

[2234] Hardware and software used

[2235] Hardware: Smartphones, smart glasses, head-mounted displays

[2236] Software: AWS (Amazon Web Services), MySQL, Python, TensorFlow, OpenCV

[2237] System Configuration

[2238] The system primarily consists of user terminals, servers, and databases. The operation and role of each major component are described in detail below.

[2239] User terminal

[2240] The user terminal uses devices such as smartphones, smart glasses, and head-mounted displays. The user terminal provides an interface for inputting information about the deceased and functions for transmitting that information. This allows users to easily input data and send it to the server.

[2241] server

[2242] The server processes information received from the user and generates a digital profile. It also uses a generative AI model to produce natural-sounding responses based on the user's input messages. The server stores the received information in a MySQL database and searches this data as needed to generate the best possible answers.

[2243] database

[2244] The database plays a role in securely storing information received from the server. The database contains encrypted data such as information about deceased individuals, user-submitted episodes, and past message histories.

[2245] Processing flow

[2246] The user first uses an interface to enter information about the deceased. This interface includes text fields, a file upload function, and a voice recording function. The information entered by the user is sent to the server, analyzed, and stored in a database. This generates a digital profile of the deceased.

[2247] When a user stands at a specific location, the server uses GPS information from their smartphone or smart glasses to retrieve specific information. Based on this, a generative AI model generates a relevant response and sends it to the user's device. The user's device then displays or plays this response as text or audio.

[2248] Emotion analysis and personalization

[2249] When a user enters a message, the sentiment analysis engine activates to recognize the user's emotions. Based on these emotions, the generative AI model generates a personalized response. For example, if a user enters "I'm really sad today," the server performs sentiment analysis and, depending on the user's emotional state, generates a comforting response such as "It's okay, I'll listen" and sends it to the user's device.

[2250] Privacy and Security

[2251] All data is end-to-end encrypted and stored in the database. If a user requests data deletion or changes to privacy settings, the server responds immediately, deleting the relevant data from the database or updating the settings. This ensures users can use the system with peace of mind.

[2252] Examples of specific cases and prompt statements

[2253] As a concrete example, if a profile of a former regular customer named "Ichiro Tanaka" is registered, and a customer at the store says using smart glasses, "I've found Ichiro's usual seat," the AI ​​will respond, "Really? Ichiro used to drink coffee here often."

[2254] Example of a prompt:

[2255] "Tell me about this place."

[2256] "Tell me about Christmas events from the past."

[2257] "Please share your memories from here."

[2258] In this way, the system provides users with a personalized, interactive experience, deepening their connection with deceased loved ones and past events. Furthermore, security measures ensure that users can use the system with peace of mind.

[2259] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2260] Step 1:

[2261] The user uses the terminal interface to enter basic information about the deceased (name, age, photo, audio sample, personal anecdotes, etc.). The entered information is sent from the user's terminal to the server. Specifically, the user enters information into a text field and clicks the "Send" button. The input is text or multimedia files (photo, audio), and the output is a data packet sent to the server.

[2262] Step 2:

[2263] The server parses the received information and stores it in a MySQL database. The server converts the input data into the appropriate format and inserts it into the database using SQL statements. The input is the user's data sent, and the output is a digital profile stored in the database.

[2264] Step 3:

[2265] When a user stands at a specific location, the server uses the device's GPS information to obtain their location. The device sends current coordinate data to the server, which then uses this data to search for relevant information. The input is the GPS data sent from the device, and the output is the relevant information returned by the server.

[2266] Step 4:

[2267] The server generates a response using a generative AI model based on relevant information. Specifically, the server searches for relevant episodes in the database and inputs them as prompts into the generative AI model. The server then analyzes the text generated by the model and produces an appropriate response. The input is relevant information based on location data, and the output is the generated response.

[2268] Step 5:

[2269] The server sends the generated response to the user's terminal, which then displays or plays it as text or audio. Specifically, the server sends the generated response to the terminal, and the terminal presents the received data to the user in an appropriate format. The input is the generated response, and the output is the text display or audio playback to the user.

[2270] Step 6:

[2271] When a user enters and sends an additional message, the server receives it and uses its sentiment analysis engine to recognize the emotion. Based on the analysis results, the server adjusts its AI model to generate a response appropriate to the emotion. The input is the user's message, and the output is the emotion-appropriate response.

[2272] Step 7:

[2273] The server encrypts all data and securely stores it in the database. The input is the data to be stored, and the output is the encrypted data. When a user requests data deletion or changes to privacy settings, the server reflects this in the database and updates the settings. Specifically, it parses the user's request and deletes or updates the corresponding data from the database. The input is the user's request, and the output is the updated database state.

[2274] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2275] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2276] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[2277] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2278] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[2279] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[2280] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[2281] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it res...

Claims

1. A means of providing an interface that allows users to input information about deceased persons, A means of sending user-entered information to the server, A means for the server to store the information it receives in a database, A means of generating a profile of a deceased person based on stored information, A means of generating a response using an AI model based on a message sent by a user, A system that includes means for sending the generated response to the user.

2. A means of providing a field for users to post episodes, A means of sending episodes entered by the user to the server, A means for the server to save the received episodes to a database, The system according to claim 1, comprising means for an AI model to generate an adapted response based on previously stored episodes.

3. A means for the server to encrypt and store all data, A means to perform the deletion of data requested by the user, The system according to claim 1, comprising means for updating the user's privacy settings.

Images