System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing a plurality of personal data associated with a deceased person or pet; means for analyzing the captured personal data to extract features of the deceased person or pet; means for generating a virtual presence of the deceased person or pet based on the extracted features; and means for providing a virtual space in which a user can interact with the generated virtual presence.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's aging society, an increasing number of people are carrying memories of their deceased loved ones and pets, as well as unresolved regrets. Furthermore, for people living in urban areas, visiting graves can be physically difficult. This creates a need for a way to virtually reconstruct relationships with deceased loved ones and pets and find peace of mind. Furthermore, by streamlining grave visits and enabling virtual interactions, it is necessary to address new needs in the end-of-life planning market. [Means for solving the problem]

[0005] The present invention is a system including: means for acquiring multiple personal data related to a deceased person or a pet; means for analyzing the acquired personal data to extract characteristics of the deceased person or pet; means for generating a virtual presence of the deceased person or pet based on the extracted characteristics; and means for providing a virtual space in which a user can interact with the generated virtual presence. The present invention also includes means for inputting personal data from a user terminal and transmitting the data to a server, and means for generating and providing a link to the virtual space to the user. This system allows users to virtually converse and interact with the deceased person or pet, reconstructing their relationship with the deceased person or pet and achieving peace of mind. Furthermore, by linking with urban indoor ossuaries and cloud cemetery operators, the system can improve the efficiency of grave visits.

[0006] "Personal data" refers to individual information, such as photographs, videos, audio, or text data, related to a deceased person or pet.

[0007] "Analysis" refers to the process of analyzing personal data and extracting characteristic information about the deceased person or pet.

[0008] "Characteristics" refers to the appearance, voice, manner of speaking, and behavioral patterns of the deceased or pet.

[0009] "Virtual presence" refers to a digital or virtual reproduction of a deceased person or pet created using a generative AI model based on analyzed characteristics.

[0010] "Virtual space" refers to a digital or online simulated space provided for users to interact with.

[0011] "User terminal" refers to a device (smartphone, tablet, PC, etc.) used to input personal data and access virtual space.

[0012] "Server" refers to the central computer system that stores personal data sent by users and processes it for analysis and generation.

[0013] "Link" refers to an online URL or similar information that allows a user to access a virtual space.

[0014] "Interaction" refers to the act of a user having conversations or engaging in actions with a deceased person or pet in a virtual space.

[0015] A "dialogue system" refers to a system that has the ability to use a generative AI model to return a response based on user input. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention provides a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system operates mainly between a server, a terminal, and users.

[0038] User Data Collection

[0039] 1. Data Collection

[0040] User: The user launches the app and selects personal data related to the deceased person or pet, including photos, videos, audio files, and text.

[0041] On your device: The device will preview the data you selected and ask you to confirm the upload. Once confirmed, the data will be sent to the server.

[0042] Server: The server stores the received data and notifies the user that it was received successfully.

[0043] Data analysis and generation

[0044] 2. Data Analysis

[0045] Server: The server analyzes the received personal data. It uses image analysis algorithms to extract features of the deceased person or pet from photos and videos, analyzes audio data to extract voice characteristics, and analyzes text data to identify frequently used words and speaking patterns.

[0046] 3. Virtual Being Creation

[0047] Server: Based on the extracted feature data, a generative AI model is used to generate a virtual presence of the deceased person or pet, including 3D model generation and voice synthesis.

[0048] Server: Stores the generated data in a database for the virtual space.

[0049] Preparing for a virtual cemetery

[0050] 4. Resource Allocation and Link Creation

[0051] Server: Allocates resources for the virtual graveyard, places the generated virtual beings in the virtual space, generates access links to the virtual space, and sends them to the terminals.

[0052] Terminal: The terminal notifies the user that the virtual cemetery is ready and provides an access link.

[0053] Dialogue and Interaction

[0054] 5. Access and Interaction

[0055] User: The user clicks on the provided access link to access the virtual cemetery. Authentication is required for access, ensuring security.

[0056] Terminal: Once access to the virtual graveyard is authenticated, an interactive interface is displayed.

[0057] 6. Start a dialogue

[0058] Server: Analyzes the message entered by the user and generates an appropriate response. For example, the question "How have you been?" will be answered by a character that resembles a deceased person or pet.

[0059] Terminal: Receives the response from the server and displays it to the user.

[0060] Specific examples

[0061] As an example, we will explain the process when a user uploads photos and videos of a deceased pet.

[0062] 1. Data Collection

[0063] User: Use the dedicated app to select and upload photos and videos of your pet.

[0064] On your device: Displays "Uploading data" and notifies you "Upload complete" once it's finished.

[0065] Server: Stores the received data and prepares it for analysis.

[0066] 2. Data Analysis and Generation

[0067] Server: Extracts pet appearance characteristics from photos and videos, and analyzes voice and behavioral patterns from videos and audio.

[0068] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[0069] 3. Preparing a Virtual Graveyard

[0070] Server: Allocates resources for the virtual cemetery and generates access links.

[0071] Terminal: Notify the user: "Your virtual graveyard is ready. Please access it at the following link: example_link."

[0072] 4. Dialogue and Interaction

[0073] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[0074] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[0075] Terminal: displays the response from the server and allows the user to continue the interaction.

[0076] The system allows users to virtually interact with deceased loved ones or pets, providing peace of mind.

[0077] The processing flow will be explained below.

[0078] Program processing steps

[0079] Step 1: Select and upload data

[0080] User: Launch the dedicated app and select photos, videos, audio, and text data of the deceased person or pet.

[0081] On the device: Preview the selected file and ask the user for confirmation. If the user presses the confirm button, the file will be uploaded to the server.

[0082] Server: Stores the received data in a database and notifies the device that the upload is complete.

[0083] Terminal: Notify the user that the upload was successful.

[0084] Step 2: Data analysis

[0085] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[0086] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[0087] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[0088] Server: Stores these feature data in a database.

[0089] Step 3: Creating a Virtual Being

[0090] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[0091] Server: Stores the generated virtual entities in a database for the virtual space.

[0092] Step 4: Preparing the Virtual Graveyard and Creating Links

[0093] Server: Allocates virtual graveyard resources and places the generated virtual beings in the virtual space.

[0094] Server: Generates a link to access the virtual cemetery and sends it to the device.

[0095] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[0096] Step 5: Access and Authentication

[0097] User: Clicks on the provided access link to access the virtual cemetery.

[0098] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[0099] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[0100] Step 6: Start a conversation

[0101] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[0102] Terminal: Sends user input to the server.

[0103] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[0104] Server: Sends the generated response to the terminal.

[0105] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[0106] Step 7: Additional Interactions

[0107] User: Continues to reminisce about and ask questions about deceased loved ones and pets. Performs specific actions in the virtual space, such as placing flowers.

[0108] Terminal: Sends user actions to the server and displays the results of the execution.

[0109] Server: Updates the virtual space in real time based on the user's actions, allowing the user to continue interacting.

[0110] In this way, users can virtually converse and interact with their deceased loved ones or pets, bringing them peace of mind.

[0111] Example 1

[0112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] In modern society, people cherish memories of their deceased loved ones and pets and wish to feel close to them forever. However, existing technologies lack the means to recreate the presence of deceased people or pets in a virtual space and provide them in an interactive form using personal data related to them. Therefore, there is a need for a system that analyzes data related to deceased people or pets, generates virtual presences of the deceased or pets based on that data, and allows users to interact with them.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0115] In this invention, the server includes: means for a user to select multiple pieces of personal data related to the deceased person or pet; means for previewing the selected personal data on a user terminal and confirming upload; means for transmitting the confirmed personal data to the server and saving it; means for analyzing the saved personal data and extracting characteristics of the deceased person or pet; means for generating a virtual being of the deceased person or pet using a generative AI model based on the extracted characteristic data; means for generating a 3D model of the generated virtual being and performing voice synthesis; means for storing the generated virtual being in a virtual space database and placing it in the virtual space; means for generating an access link to the virtual space and sending it to the user terminal; means for a user to access the virtual space and be authenticated by entering authentication information; means for enabling interaction with the virtual being in the virtual space if authentication is successful; means for analyzing messages from the interacting user and generating appropriate responses; and means for displaying the generated responses on the user terminal. This allows users to comprehensively organize data related to the deceased person or pet, generate virtual beings based on the data, and enjoy interacting with them.

[0116] "User" refers to a person who uses the system to manipulate personal data related to a deceased person or pet and to interact in a virtual space.

[0117] "Personal data" refers to digital information such as photos, videos, audio files, and text related to a deceased person or pet.

[0118] "User terminal" refers to a device used by a user to select, preview, review, and transmit personal data.

[0119] "Server" refers to a central computer system that receives, stores, and analyzes personal data sent by users and generates a virtual presence.

[0120] An "image analysis algorithm" refers to a computational method for extracting specific features from photos and videos.

[0121] "Speech analysis" refers to the process of extracting speech features from speech data.

[0122] "Text analytics" refers to the process of analyzing text data for frequently used words and speech patterns.

[0123] "Generative AI model" refers to an AI (artificial intelligence) model that uses machine learning technology to generate virtual versions of deceased people or pets.

[0124] "3D model generation" refers to the process of creating a three-dimensional virtual model based on characteristic data of a deceased person or pet.

[0125] "Speech synthesis" refers to the process of generating new speech from analyzed speech data.

[0126] "Database for virtual space" refers to a database for storing data on generated virtual entities.

[0127] "Access link" refers to a URL or the like that allows a user to access a virtual space.

[0128] "Authentication" refers to the process of verifying a user's identity when they access a system.

[0129] "Interaction interface" refers to a display and input system that allows a user and a virtual entity to interact in a virtual space.

[0130] "Prompt sentence" refers to an input sentence provided to an AI model to elicit an appropriate response.

[0131] "Response" refers to the appropriate response that a generative AI model gives in response to user input.

[0132] MODE FOR CARRYING OUT THE INVENTION

[0133] The present invention relates to a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system operates mainly between a server, a terminal, and users.

[0134] First, the user launches the app and selects personal data such as photos, videos, audio files, and text related to the deceased person or pet. The device previews the data and asks for confirmation of upload. Once confirmed, the data is sent from the user's device to the server using the HTTPS protocol. The server stores the received data and notifies the user that it has been successfully received.

[0135] The server then analyzes the stored data, which includes the following steps:

[0136] 1. Image analysis: Use OpenCV to extract facial landmark points and other features from photos and videos.

[0137] 2. Audio analysis: Using the LibROSA library, Mel-Frequency Cepstrum Coefficients (MFCCs) are extracted from the audio data and saved as feature data.

[0138] 3. Text Analysis: Use the NLTK library to tokenize text data and analyze it for frequently occurring words and phrases.

[0139] The server then uses a generative AI model (e.g., GPT-3) based on the extracted feature data to generate a virtual representation of the deceased person or pet, including:

[0140] 1. 3D model generation: Using Blender's API, a 3D model is automatically generated based on facial feature data.

[0141] 2. Voice synthesis: Voice data is generated based on the voice characteristics analyzed by Tacotron 2 and used for virtual characters.

[0142] The generated data is stored in a database for the virtual space, and the server allocates resources for the virtual cemetery and places the generated virtual beings in the virtual space. An access link to the virtual space is generated and sent to the user's terminal. The user's terminal notifies the user that the virtual cemetery is ready and provides the access link.

[0143] When a user clicks on the provided access link, they are given access to the virtual cemetery. Authentication is required for access, and the user enters their authentication information. Once access is authenticated, an interactive interface is displayed.

[0144] When a conversation begins, the server analyzes the message entered by the user and generates an appropriate response. For example, if the user asks, "How have you been?", a response that sounds like the character of the deceased person or pet will be returned (e.g., "I've been fine! How have you been?"). The response from the server is displayed on the user's device, allowing the user to continue the conversation.

[0145] Specific examples

[0146] Data collection

[0147] User: Use the dedicated app to select and upload photos and videos of your pet.

[0148] On your device: Displays "Uploading data" and notifies you "Upload complete" once it's finished.

[0149] Server: Stores the received data and prepares it for analysis.

[0150] Data analysis and generation

[0151] Server: Extracts pet appearance characteristics from photos and videos, and analyzes voice and behavioral patterns from videos and audio.

[0152] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[0153] Preparing for a virtual cemetery

[0154] Server: Allocates resources for the virtual cemetery and generates access links.

[0155] Terminal: Notify the user: "Your virtual graveyard is ready. Please access it at the following link: example_link."

[0156] Dialogue and Interaction

[0157] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[0158] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[0159] Terminal: displays the response from the server and allows the user to continue the interaction.

[0160] As described above, this invention allows users to comprehensively organize data related to deceased loved ones and pets, and based on that data, generate virtual beings and enjoy conversations with them. This system allows users to virtually converse with deceased loved ones and pets, providing peace of mind.

[0161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0162] Step 1: The user launches the dedicated app and selects personal data (photos, videos, audio files, text) related to the deceased person or pet.

[0163] Input: The user operates the dedicated app and specifies personal data.

[0164] Specific operation: The user uses a smartphone or PC to select the relevant file from a folder using the "Data Selection" function within the app.

[0165] Step 2: Your device will preview the selected personal data and ask you to confirm the upload.

[0166] Input: A data file selected by the user.

[0167] Specific behavior: The device will display a preview window to allow the user to check the contents of the data, and a confirmation dialog will appear with the message "Do you want to upload?"

[0168] Step 3: The user confirms the upload and the device sends the data to the server.

[0169] Input: User upload confirmation.

[0170] Output: The data sent to the server.

[0171] What it does: When the user presses the "Upload" button, the device compresses the selected data and sends it to the server using the HTTPS protocol.

[0172] Step 4: The server stores the received data and notifies the user of successful reception.

[0173] Input: Data from the terminal.

[0174] Output: Saved data and notification to the user.

[0175] Specific operation: The server stores the received data in a database and returns an HTTP response saying "Data upload completed." The device receives this response and notifies the user.

[0176] Step 5: The server analyzes the received personal data.

[0177] Input: Stored personal data.

[0178] Output: Extracted feature data.

[0179] Specific behavior:

[0180] Image analysis: Extract facial landmark points from photos and videos using OpenCV.

[0181] Audio analysis: Extract Mel-Frequency Cepstral Coefficients (MFCCs) from audio files using the LibROSA library.

[0182] Text analysis: Use the NLTK library to tokenize text data and analyze frequently occurring words and phrases.

[0183] Step 6: The server uses the extracted feature data to generate a virtual presence of the deceased person or pet using a generative AI model.

[0184] Input: Extracted feature data.

[0185] Output: Data of the generated virtual entity.

[0186] Specific behavior:

[0187] A generative AI model (e.g., GPT-3) is trained on collected text patterns to generate hypothetical conversational responses.

[0188] Using Blender's API, a 3D model is generated based on facial feature data.

[0189] Voice data is generated based on the voice characteristics analyzed by Tacotron 2.

[0190] Step 7: The server stores the generated data in a database for the virtual space and places it in the virtual space.

[0191] Input: Data of the generated virtual entity.

[0192] Output: A virtual entity placed in virtual space.

[0193] How it works: The server uses the Unity server to assign virtual characters to virtual spaces and save the corresponding metadata in a database.

[0194] Step 8: The server generates an access link to the virtual space and sends it to the terminal.

[0195] Input: Virtual space resource information.

[0196] Output: Access link.

[0197] Specific operation: The server returns a dynamically generated URL link to the device in an HTTP response, and the device provides the link to the user.

[0198] Step 9: The user clicks on the provided access link to access the virtual cemetery, enters their authentication information and is authenticated.

[0199] Input: Access link, authentication information.

[0200] Output: Authenticated access to the virtual space.

[0201] Specific operation: The user opens the access link in a browser and enters their user ID and password on the displayed login screen. The server performs authentication, and if successful, they enter the virtual space.

[0202] Step 10: The user inputs a message into the dialogue interface in the virtual space, and the server generates an appropriate response.

[0203] Input: The user's message.

[0204] Output: The appropriate response.

[0205] Specific operation: The server receives the user's message, sends a prompt such as "The user is asking how are you?" to the GPT-3 model, and receives the generated response.

[0206] Step 11: The server sends the generated response to the user terminal and displays it to the user.

[0207] Input: The server-generated response.

[0208] Output: The response displayed on the user's terminal.

[0209] Specific operation: The server sends the generated response as an HTTP response to the user's terminal, and the terminal displays it to the user in text or audio format.

[0210] (Application example 1)

[0211] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0212] Previous virtual presence systems related to deceased loved ones or pets lacked the interactive services that users could experience directly in a physical store. This prevented users from feeling closer to their loved ones or pets through virtual interactions, making it difficult to fully experience peace of mind. Furthermore, without incorporating advanced AI models to generate natural interactions, the user experience was unnatural and did not feel as close to real interactions.

[0213] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0214] In this invention, the server includes means for acquiring a plurality of personal data related to the deceased person or pet, means for analyzing the acquired personal data and extracting characteristics of the deceased person or pet, means for generating a virtual presence of the deceased person or pet based on the extracted characteristics, means for providing an interaction interface that allows a user to interact with the virtual presence in a physical store, and means for using a generative AI model to generate natural interactions when the user interacts with the virtual presence, thereby enabling users to interact with the deceased person or pet in a natural way in a physical store and gain peace of mind.

[0215] "Personal data" refers to information such as photos, videos, audio files, and text related to an individual.

[0216] "Features" refer to attributes of the deceased person or pet, such as their appearance, voice, or language usage patterns, extracted from image, audio, and text data.

[0217] "Virtual presence" refers to a virtual character that can be interacted with, including a 3D model and voice, generated from the personal data of a deceased person or pet.

[0218] "Interaction interface" refers to a device, such as a tablet, robot, or smart glasses, that a user uses to communicate with a virtual being.

[0219] A "generative AI model" refers to an artificial intelligence model that generates natural responses to user input.

[0220] "Brick and mortar store" refers to a physical store that a user can actually visit.

[0221] The present invention provides a system that allows users to experience virtual interactions with deceased loved ones or pets in a physical store. This system provides a comprehensive service, from collecting personal data to generating virtual beings and interacting with them through an interface. Specific embodiments are described below.

[0222] Data collection

[0223] Users use dedicated terminals (tablets, robots, or smart glasses) installed in physical stores to input personal data related to the deceased person or pet. This personal data includes photos, videos, audio files, and text. The dedicated terminals then send this data to a server.

[0224] Data analysis and virtual entity generation

[0225] The server analyzes the received personal data. Image data is extracted using DeepFace to identify physical features of the deceased person or pet, while audio data is extracted using Google Text-to-Speech (gTTS) and other voice analysis tools. Text data is analyzed using the OpenAI API to identify frequently used words and speech patterns.

[0226] Based on this feature data, a generative AI model is used to generate a virtual entity, which includes a 3D model and a synthesized voice, and is placed in a virtual space.

[0227] Providing a conversational interface

[0228] The dedicated terminal provides the user with a dialogue interface through which the user can interact with the virtual entity. Questions and comments entered by the user are sent to the server.

[0229] The server analyzes the user's input using a generative AI model and generates an appropriate response. The response is designed to be natural and takes into account the characteristics of the deceased person or pet. The generated response is sent to a dedicated terminal and displayed to the user.

[0230] Specific examples

[0231] For example, if a user uploads photos and videos of their beloved dog "Pochi" in a physical store and asks, "Pochi, how have you been lately?", the server analyzes this data and generates a virtual Pochi. Using a generative AI model, it generates a response to the question, "Pochi, how have you been lately?", returning the response, "I've been fine! How have you been?"

[0232] Prompt Sentence Examples

[0233] Below are some examples of prompt sentences.

[0234] Please reply as Pochi to the following questions: What have you been up to lately?

[0235] The system thus configured allows users to interact with virtual deceased loved ones or pets through natural conversations within a physical store, providing peace of mind.

[0236] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0237] Step 1:

[0238] Users use dedicated terminals (tablets, robots, or smart glasses) installed in physical stores to input personal data such as photos, videos, audio files, and text related to the deceased person or pet. The input data is sent to the server by the terminal. The input is completed when the user selects the data and presses the "upload" button.

[0239] Step 2:

[0240] The server stores the personal data received from the device and then begins analyzing it. Specifically, image data from photos and videos is used to extract physical features of the deceased person or pet using DeepFace. Voice data is used to extract features using Google Text-to-Speech (gTTS) and other voice analysis tools. Text data is analyzed using the OpenAI API to identify frequently used words and speaking patterns.

[0241] Step 3:

[0242] The server uses a generative AI model to generate a virtual presence (3D model and voice) based on the feature data extracted in step 2. This virtual presence reflects the unique characteristics of the deceased person or pet and is capable of natural conversation. The generated virtual presence is stored in a database for the virtual space.

[0243] Step 4:

[0244] The dedicated terminal provides a dialogue interface to the user. The user can use this interface to start a dialogue with the virtual entity. For example, when the user types "Pochi, how have you been lately?", the data is sent from the terminal to the server.

[0245] Step 5:

[0246] The server analyzes the user's questions and comments received in step 4 and generates an appropriate response using the generative AI model. This response is designed to be a natural conversation that takes into account the characteristics of the deceased person or pet. For example, in response to a user's question, "Pochi, how have you been lately?", the server generates a response such as, "I've been fine! How have you been?"

[0247] Step 6:

[0248] The generated response is sent from the server to a dedicated terminal and displayed to the user. The user can confirm the response and continue the dialogue. The content of the dialogue is recorded on the server and used as reference for future dialogues.

[0249] Step 7:

[0250] When the user finishes the conversation, the dedicated terminal closes the conversation interface and the server ends the session. If the user wishes, the conversation content can be saved and a link for the next access can be provided.

[0251] Through the above steps, users can have natural conversations with their deceased loved ones or pets in a physical store, providing peace of mind.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] The present invention is a system that allows users to create virtual beings based on personal data related to deceased people or pets and engage in conversations in a virtual space. This system incorporates an emotion engine that analyzes the user's emotions, enabling more personalized conversations. The system operates primarily between a server, a terminal, and the user.

[0254] User Data Collection

[0255] 1. Data Collection

[0256] User: The user launches the app and selects personal data related to the deceased person or pet, including photos, videos, audio files, and text.

[0257] On the device: The device will preview the data you selected and ask you to confirm the upload. Once confirmed, the data will be uploaded to the server.

[0258] Server: The server stores the received data and notifies the user that it was received successfully.

[0259] Data analysis and generation

[0260] 2. Data Analysis

[0261] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[0262] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[0263] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[0264] Server: Stores these feature data in a database.

[0265] 3. Virtual Being Creation

[0266] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[0267] Server: Stores the generated virtual entities in a database for the virtual space.

[0268] Preparing for a virtual cemetery

[0269] 4. Resource Allocation and Link Creation

[0270] Server: Allocates resources for the virtual graveyard and places the generated virtual beings in the virtual space.

[0271] Server: Generates a link to access the virtual cemetery and sends it to the device.

[0272] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[0273] Use of dialogue and emotion engines

[0274] 5. Access and Authentication

[0275] User: Clicks on the provided access link to access the virtual cemetery.

[0276] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[0277] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[0278] 6. Start a dialogue

[0279] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[0280] Terminal: Sends user input to the server.

[0281] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[0282] Server: Sends the generated response to the terminal.

[0283] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[0284] Use of emotion engine

[0285] 7. Emotional Recognition

[0286] Device: The device is equipped with facial recognition and voice analysis functions, and analyzes the user's emotions in real time from their facial expressions and voice.

[0287] Server: Receives the analysis results and sends them to the emotion engine.

[0288] Server: The emotion engine adjusts the content and tone of the response based on the user's emotional data.

[0289] 8. Personalized responses

[0290] Server: Generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it will return words of encouragement.

[0291] Server: Sends the generated response to the terminal.

[0292] Terminal: Displays the response received from the server to the user, for example, "Cheer up, we're always thinking about you."

[0293] Specific examples

[0294] As an example, we will explain how a user can upload photos or videos of a deceased pet and start a dialogue using the emotion engine.

[0295] 1. Data Collection

[0296] User: Use the dedicated app to select and upload photos and videos of your pet.

[0297] On your device: Display the progress of the data upload and notify you when it's finished by saying "Upload complete."

[0298] Server: Stores the received data and prepares it for analysis.

[0299] 2. Data Analysis and Generation

[0300] Server: Extracts the pet's appearance characteristics from photos and videos and analyzes its voice characteristics.

[0301] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[0302] 3. Preparing a Virtual Graveyard

[0303] Server: Allocates resources for the virtual cemetery and generates access links.

[0304] Terminal: Send the link to the user and tell them, "Your virtual cemetery is ready. Please access it from the following link: example_link."

[0305] 4. Use of dialogue and emotion engines

[0306] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[0307] Terminal: Sends user input to the server.

[0308] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[0309] Terminal: Displays the response from the server and shows it to the user.

[0310] Terminal: Analyzes emotions from the user's facial expressions and voice, and sends the results to the server.

[0311] Server: The emotion engine generates a response based on the user's emotions, for example, returning encouraging words such as "You're sad, we're thinking of you too."

[0312] The system allows users to virtually interact with deceased loved ones or pets and receive personalized responses based on their emotions, providing greater peace of mind.

[0313] The processing flow will be explained below.

[0314] Program processing steps

[0315] Step 1: Select and upload data

[0316] User: Launch the dedicated app and select photos, videos, audio, and text data of the deceased person or pet.

[0317] On the device: Preview the selected file and ask the user for confirmation. If the user presses the confirm button, the file will be uploaded to the server.

[0318] Server: Stores the received data in a database and notifies the device that the upload is complete.

[0319] Terminal: Notify the user that the upload was successful.

[0320] Step 2: Data analysis

[0321] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[0322] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[0323] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[0324] Server: Stores these feature data in a database.

[0325] Step 3: Creating a Virtual Being

[0326] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[0327] Server: Stores the generated virtual entities in a database for the virtual space.

[0328] Step 4: Preparing the Virtual Graveyard and Creating Links

[0329] Server: Allocates resources for the virtual graveyard and places the generated virtual beings in the virtual space.

[0330] Server: Generates a link to access the virtual cemetery and sends it to the device.

[0331] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[0332] Step 5: Access and Authentication

[0333] User: Clicks on the provided access link to access the virtual cemetery.

[0334] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[0335] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[0336] Step 6: Start a conversation

[0337] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[0338] Terminal: Sends user input to the server.

[0339] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[0340] Server: Sends the generated response to the terminal.

[0341] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[0342] Step 7: Recognize emotions

[0343] Device: The device is equipped with facial recognition and voice analysis functions, and analyzes the user's emotions in real time from their facial expressions and voice.

[0344] Server: Receives the analysis results and sends them to the emotion engine.

[0345] Server: The emotion engine adjusts the content and tone of the response based on the user's emotional data.

[0346] Step 8: Personalize your response

[0347] Server: Generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it will return words of encouragement.

[0348] Server: Sends the generated response to the terminal.

[0349] Terminal: Displays the response received from the server to the user, for example, "Cheer up, we're always thinking about you."

[0350] Step 9: Additional Interactions

[0351] User: Continues to reminisce about and ask questions about deceased loved ones and pets. Performs specific actions in the virtual space, such as placing flowers.

[0352] Terminal: Sends user actions to the server and displays the results of the execution.

[0353] Server: Updates the virtual space in real time based on the user's actions, allowing the user to continue interacting.

[0354] In this way, users can virtually converse and interact with their deceased loved ones or pets, bringing them peace of mind.

[0355] Example 2

[0356] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] Today, many people use photos and videos to remember their deceased loved ones and pets, but these alone are not enough to bring them peace of mind. While there are systems that allow users to virtually interact with their deceased loved ones and pets, few systems can respond to emotional changes in real time. As a result, it is currently difficult for users to receive personalized experiences tailored to their emotions.

[0358] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to select multiple pieces of personal data related to the deceased person or pet; means for transmitting the selected personal data to the server; means for analyzing the personal data received by the server and extracting characteristics of the deceased person or pet; means for generating a virtual presence of the deceased person or pet using a generative AI model based on the extracted characteristics; means for placing the generated virtual presence in a virtual space; means for generating and providing a link for accessing the virtual space to the user; means for the user to interact with the virtual presence in the virtual space; an emotion engine for analyzing the user's emotions in real time; and means for personalizing responses based on the emotion engine. This allows the user to receive personalized responses according to their emotions while virtually interacting with the deceased person or pet.

[0359] "User" refers to an individual who uses this system.

[0360] "Personal data" refers to information such as photos, videos, audio files, and text related to the deceased person or pet.

[0361] "Terminal" refers to a device operated by a user, such as a computer, smartphone, or tablet.

[0362] "Server" refers to the computer that processes and stores data at the center of this system.

[0363] "Analysis" refers to the process of extracting characteristics and information about the deceased person or pet from personal data.

[0364] A "generative AI model" refers to an artificial intelligence model that uses machine learning algorithms to generate new data or formats.

[0365] "Virtual presence" refers to a digital representation of a deceased person or pet created by a generative AI model based on analyzed personal data.

[0366] "Virtual space" refers to a digital environment in which users interact with virtual beings.

[0367] "Link" refers to a URL or hyperlink for accessing a virtual space.

[0368] An "emotion engine" refers to a system that analyzes a user's emotions and generates a response based on those emotions.

[0369] The present invention provides a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system is mainly composed of a server, terminals, and users.

[0370] User Data Collection

[0371] User: Launches the dedicated app and selects photos, videos, audio files, and text files related to the deceased person or pet.

[0372] On the device: Preview the selected data and ask the user to confirm the upload. Once confirmed, send the data to the server.

[0373] Server: Saves the received data in local storage or cloud storage and notifies the user that the upload is complete.

[0374] Data analysis and generation

[0375] Server: Analyzes the received photos and videos using image analysis algorithms (e.g., OpenCV or TensorFlow) to extract the external characteristics of the deceased person or pet.

[0376] Server: The audio file is analyzed using a speech analysis algorithm (e.g., Google Cloud Speech-to-Text or IBM Watson) to extract voice characteristics.

[0377] Server: Analyzes the text data using natural language processing algorithms (e.g., spaCy or BERT) to extract frequently used words and distinctive language patterns.

[0378] Server: Stores these feature data in a database (e.g., PostgreSQL or MongoDB).

[0379] Server: Based on the stored feature data, a generative AI model (e.g., GPT-3 or StyleGAN) is used to generate virtual deceased people or pets. This generation process includes creating 3D models and synthesizing voices. The generated virtual beings are stored in a database for the virtual space.

[0380] Preparing for a virtual cemetery

[0381] Server: Allocate resources for the virtual cemetery and place the virtual presence of deceased people and pets in the virtual space. This includes securing cloud server resources and generating the virtual space.

[0382] Server: Generates and provides users with a dedicated link to access the virtual cemetery.

[0383] Terminal: Notifies the user of the generated access link and informs them that the virtual cemetery is ready.

[0384] Use of dialogue and emotion engines

[0385] User: Clicks on the provided access link to access the virtual cemetery.

[0386] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[0387] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[0388] User: Initiate a virtual conversation with a deceased loved one or pet, for example, by typing "How have you been?"

[0389] Terminal: Sends user input to the server.

[0390] Server: Analyzes the user's input using an AI model, generates responses that the deceased person or pet would likely make, and sends them to the device.

[0391] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[0392] Use of emotion engine

[0393] Device: Using facial recognition and voice analysis functions, emotions are analyzed in real time from the user's facial expressions and voice.

[0394] Server: Receives the analysis results and sends them to the emotion engine.

[0395] Server: The emotion engine generates personalized responses by adjusting the content and tone of the response based on the user's emotional data. For example, if the user is sad, it adds words of encouragement.

[0396] Server: Sends the generated response to the terminal and displays it to the user, for example, "Cheer up, I'm always worried about you."

[0397] Specific examples

[0398] For example, consider a user uploading photos and videos of a deceased pet to use the system:

[0399] 1. User: Use the dedicated app to select and upload photos and videos of your pet.

[0400] 2. Device: Displays the progress of data upload and notifies you when it is complete.

[0401] 3. Server: Stores the received data and prepares it for analysis.

[0402] 4. Server: Extracts the pet's external features from photos and videos and analyzes its audio features.

[0403] 5. Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[0404] 6. Server: Allocates resources for the virtual cemetery and generates access links.

[0405] 7. Terminal: Notify the user of the link and say, "Your virtual cemetery is ready. Please access it from the following link: example_link."

[0406] 8. User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[0407] 9. Terminal: Sends user input to the server and displays responses to the user. It displays responses such as "Pochi was fine! How was yours?"

[0408] 10. Terminal: Analyzes emotions from the user's facial expressions and voice and sends them to the server.

[0409] 11. Server: The emotion engine generates a response based on the user's emotions, returning encouraging words such as "You're sad, we're thinking of you too." This system allows the user to find greater peace of mind.

[0410] This Detailed Description provides a detailed understanding of how the present invention may be practiced.

[0411] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0412] Program processing steps

[0413] Step 1: Collect user data

[0414] Input: The user selects photos, videos, audio files, and text related to the deceased person or pet.

[0415] How it works: Launch the dedicated app and select data through the interface.

[0416] Output: The selected data is saved to the device and sent to the next processing step.

[0417] Step 2: Upload data

[0418] Input: Personal data stored on your device.

[0419] How it works: The device previews the selected data and asks the user to confirm the upload. Once confirmed, the data is sent to the server.

[0420] Output: Personal data is uploaded to the server.

[0421] Step 3: Data storage and notification

[0422] Input: Personal data received by the server.

[0423] How it works: The server saves the received data to local storage or cloud storage and notifies the user that the upload is complete.

[0424] Output: The data is saved and the user is notified that the upload is complete.

[0425] Step 4: Image analysis

[0426] Input: Photos and videos from your saved personal data.

[0427] How it works: The server uses image analysis algorithms (e.g., OpenCV or TensorFlow) to extract the external features of the deceased person or pet.

[0428] Output: Appearance feature data is generated.

[0429] Step 5: Audio analysis

[0430] Input: Audio file from saved personal data.

[0431] How it works: The server uses voice analysis algorithms (e.g., Google Cloud Speech-to-Text or IBM Watson) to extract voice characteristics and tone.

[0432] Output: Voice feature data is generated.

[0433] Step 6: Text Analysis

[0434] Input: A text file from your saved personal data.

[0435] How it works: The server uses natural language processing algorithms (e.g. spaCy or BERT) to analyze frequently used words and unique language patterns.

[0436] Output: Word and language pattern data is generated.

[0437] Step 7: Save to database

[0438] Input: Analyzed appearance feature data, voice feature data, and word and language pattern data.

[0439] How it works: The server stores this data in a database (e.g. PostgreSQL or MongoDB).

[0440] Output: Feature data stored in a database.

[0441] Step 8: Virtual Being Generation

[0442] Input: Stored feature data.

[0443] How it works: The server uses generative AI models (e.g., GPT-3 or StyleGAN) to generate virtual versions of deceased people and pets, including creating 3D models and synthesizing voices.

[0444] Output: Virtual entity data is generated.

[0445] Step 9: Virtual Space Placement

[0446] Input: Data of the generated virtual being.

[0447] How it works: The server allocates resources for the virtual cemetery and places virtual figures of deceased people and pets in the virtual space.

[0448] Output: A virtual presence of a deceased person or pet placed in a virtual space.

[0449] Step 10: Link generation and notification

[0450] Input: A virtual entity placed in a virtual space.

[0451] How it works: The server generates a dedicated link to access the virtual cemetery and provides it to the user.

[0452] Output: The generated access link is notified to the user.

[0453] Step 11: Authentication and Access

[0454] Input: The access link the user received.

[0455] How it works: The user clicks on the provided link to access the virtual cemetery. The device sends authentication information to the server, which then performs the authentication process.

[0456] Output: The user can now access the virtual graveyard.

[0457] Step 12: Start a conversation

[0458] Input: A prompt that the user types in the virtual space (e.g., "How have you been?").

[0459] How it works: The device sends user input to the server, which then analyzes the prompt and uses a generative AI model to generate a response for the deceased person or pet. The generated response is then sent to the device.

[0460] Output: The user will see a response from the deceased person or pet (e.g., "I'm fine! How have you been?").

[0461] Step 13: Emotion Recognition

[0462] Input: The user's facial expressions and voice during the conversation.

[0463] How it works: The device uses facial recognition and voice analysis to analyze the user's facial expressions and voice in real time.

[0464] Output: The parsed emotion data is generated.

[0465] Step 14: Personalize your response

[0466] Input: Parsed emotion data and the following user input:

[0467] Operation: The server uses the emotion engine to generate a response based on the user's emotion. The generated response is sent to the device.

[0468] Output: A personalized response is displayed to the user (e.g., "Cheer up, we're always worried about you").

[0469] This allows the user to enjoy a virtual conversation while receiving individual responses according to their emotions.

[0470] (Application example 2)

[0471] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0472] Conventional virtual dialogue systems have the problem that interactions with virtual beings generated based on personal data related to deceased loved ones or pets cannot flexibly respond to the user's emotions or circumstances, which limits the peace of mind that users can gain and makes it difficult to achieve personalized dialogue.

[0473] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for importing multiple pieces of personal data related to the deceased person or pet, means for analyzing the imported personal data and extracting characteristics of the deceased person or pet, means for generating a virtual presence of the deceased person or pet based on the extracted characteristics, means for providing a virtual space in which the user can interact with the generated virtual presence, means including an emotion engine for analyzing the user's emotions, and means for adjusting the content and tone of the response based on the analyzed emotion data. This enables personalized dialogue according to the user's emotions and situation.

[0474] "Deceased person or pet" refers to an object that the user recognizes as a deceased person or pet and recreates in the virtual space.

[0475] "Personal Data" refers to data containing personal information, such as photographs, videos, audio files, and text related to a deceased person or pet.

[0476] "Means for extracting features" refers to the processes and techniques for analyzing and extracting the appearance, vocal characteristics, and language patterns of the deceased person or pet from the collected personal data.

[0477] "Means for generating virtual entities" refers to the technology and process for generating virtual entities, such as 3D models and voices of deceased individuals or pets, based on extracted feature data.

[0478] "Virtual space" refers to a digitally simulated environment in which users can interact with generated virtual beings.

[0479] An "emotion engine" refers to software and algorithms that analyze a user's emotions from their facial expressions and voice and adjust the content and tone of their responses accordingly.

[0480] "Means for providing an interactive virtual space" refers to the technologies and processes for constructing and making accessible a virtual space that allows users to interact with generated virtual beings.

[0481] "Emotion data" refers to information about emotions obtained by analyzing the user's facial expressions and voice.

[0482] "Response content" refers to the content of a reply to an input made by a user to a virtual entity.

[0483] "Tone of response" refers to the nuances and tone of voice of the response to the user that are adjusted by the emotion engine.

[0484] An embodiment of the present invention will be described in detail below. The present invention is a system that allows a user to generate a virtual entity based on personal data related to a deceased person or pet and to engage in a dialogue in a virtual space. The system incorporates an emotion engine and includes a function for analyzing the user's emotions and providing a more personalized dialogue.

[0485] Users use a dedicated application to collect personal data related to the deceased person or pet. This application is installed on a smartphone and allows users to upload data such as photos, videos, audio files, and text. The device previews the data selected by the user, asks for confirmation of the upload, and once confirmation is received, sends the data to the server. The server stores the received data and notifies the user that it has been successfully received.

[0486] Using the generative AI model, the server analyzes the received data and extracts the characteristics of the deceased person or pet. Specifically, it uses an image analysis algorithm to extract external features from photos and videos, and an audio analysis algorithm to extract voice characteristics and tone from audio data. It also analyzes frequently used words and unique language patterns from text data and stores this feature data in a database. Based on the analyzed feature data, the server creates a 3D model and synthesizes voice to generate a virtual deceased person or pet.

[0487] The generated virtual beings are placed in a virtual space. The server allocates resources for the virtual space, places the generated virtual beings in the virtual space, and generates a link to access the virtual cemetery. This link is provided to the user via the terminal, and the user clicks on the provided link to access the virtual cemetery. The terminal sends the user's authentication information to the server, and if the authentication is successful, a dialogue system is launched and guides the user to the virtual cemetery.

[0488] Users can interact with their deceased loved ones or pets in a virtual space. For example, by inputting a question such as "How have you been?", the device sends the input to the server, which then analyzes it using a generative AI model to generate a response that the deceased or pet would likely give. The generated response is then sent to the device and displayed to the user.

[0489] The device also provides a function that analyzes the user's emotions in real time using an emotion engine. The device is equipped with facial recognition and voice analysis functions, which analyze the user's emotions from their facial expressions and voice and send the results to the server. The server then uses the emotion engine to adjust the content and tone of the response based on the analysis results, generating a personalized response that matches the user's emotions.

[0490] Here are some examples of prompts:

[0491] User: Pochi, what have you been up to lately?

[0492] System: Pochi was fine! How were you?

[0493] The system allows users to virtually interact with deceased loved ones or pets and receive personalized responses based on their emotions, providing greater peace of mind.

[0494] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0495] Step 1:

[0496] The user launches a dedicated application and selects personal data (photos, videos, audio files, text, etc.) related to the deceased person or pet. The application prompts the user to upload the selected data. The application previews the data selected by the user and asks for confirmation of the upload. Once the user confirms, the device sends the data to the server. The input is the personal data selected by the user, and the output is the transmission of the data to the server.

[0497] Step 2:

[0498] The server stores the received personal data and notifies the user that it has been received successfully. The server then uses image analysis algorithms to analyze the photos and videos to extract the physical characteristics of the deceased person or pet. It also uses voice analysis algorithms to analyze the audio data to extract voice characteristics and tone. It also extracts frequently used words and distinctive language patterns from the text data. The input is the personal data uploaded to the server, and the output is data on physical characteristics, voice characteristics, and language patterns.

[0499] Step 3:

[0500] The server uses a generative AI model to generate a virtual deceased person or pet based on the analyzed feature data. This generation process includes creating a 3D model and synthesizing voice. The data of the generated virtual being is stored in a database for the virtual space. The input is the extracted feature data, and the output is the data of the generated virtual being.

[0501] Step 4:

[0502] The server allocates resources for the virtual cemetery and places the generated virtual beings in the virtual space. Next, it generates a link to access the virtual cemetery and sends it to the terminal. The terminal notifies the user of this link and informs them that the virtual cemetery is ready. The input is the data of the virtual beings, and the output is the generation and notification of the access link.

[0503] Step 5:

[0504] The user clicks on the provided access link to access the virtual cemetery. The terminal sends the user's authentication information to the server and executes the authentication process. If the authentication is successful, the server launches a dialogue system and guides the user to the virtual cemetery. The input is the user's access link and authentication information, and the output is access to the virtual cemetery and the launch of the dialogue system.

[0505] Step 6:

[0506] The user interacts with the deceased person or pet in the virtual space. For example, they might type, "How have you been?" The device sends the user's input to the server. The server analyzes the user's input using a generative AI model and generates a response that the deceased person or pet would likely give. The generated response is sent to the device and displayed to the user. The input is the user's interaction, and the output is the generated response.

[0507] Step 7:

[0508] The device uses facial recognition and voice analysis functions to analyze emotions from the user's facial expressions and voice. The analysis results are sent to the server. The server's emotion engine adjusts the response content and tone based on the user's emotional data. The input is the user's facial expressions and voice data, and the output is the adjusted response content and tone.

[0509] Step 8:

[0510] The server generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it generates words of encouragement and sends the generated response to the device, which then displays it to the user. The input is the analysis results of the emotion engine, and the output is a personalized response.

[0511] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0512] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0513] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0514] [Second embodiment]

[0515] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0516] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0517] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0518] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0519] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0520] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0521] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0522] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0523] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0524] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0525] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0526] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0527] The present invention provides a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system operates mainly between a server, a terminal, and users.

[0528] User Data Collection

[0529] 1. Data Collection

[0530] User: The user launches the app and selects personal data related to the deceased person or pet, including photos, videos, audio files, and text.

[0531] On your device: The device will preview the data you selected and ask you to confirm the upload. Once confirmed, the data will be sent to the server.

[0532] Server: The server stores the received data and notifies the user that it was received successfully.

[0533] Data analysis and generation

[0534] 2. Data Analysis

[0535] Server: The server analyzes the received personal data. It uses image analysis algorithms to extract features of the deceased person or pet from photos and videos, analyzes audio data to extract voice characteristics, and analyzes text data to identify frequently used words and speaking patterns.

[0536] 3. Virtual Being Creation

[0537] Server: Based on the extracted feature data, a generative AI model is used to generate a virtual presence of the deceased person or pet, including 3D model generation and voice synthesis.

[0538] Server: Stores the generated data in a database for the virtual space.

[0539] Preparing for a virtual cemetery

[0540] 4. Resource Allocation and Link Creation

[0541] Server: Allocates resources for the virtual graveyard, places the generated virtual beings in the virtual space, generates access links to the virtual space, and sends them to the terminals.

[0542] Terminal: The terminal notifies the user that the virtual cemetery is ready and provides an access link.

[0543] Dialogue and Interaction

[0544] 5. Access and Interaction

[0545] User: The user clicks on the provided access link to access the virtual cemetery. Authentication is required for access, ensuring security.

[0546] Terminal: Once access to the virtual graveyard is authenticated, an interactive interface is displayed.

[0547] 6. Start a dialogue

[0548] Server: Analyzes the message entered by the user and generates an appropriate response. For example, the question "How have you been?" will be answered by a character that resembles a deceased person or pet.

[0549] Terminal: Receives the response from the server and displays it to the user.

[0550] Specific examples

[0551] As an example, we will explain the process when a user uploads photos and videos of a deceased pet.

[0552] 1. Data Collection

[0553] User: Use the dedicated app to select and upload photos and videos of your pet.

[0554] On your device: Displays "Uploading data" and notifies you "Upload complete" once it's finished.

[0555] Server: Stores the received data and prepares it for analysis.

[0556] 2. Data Analysis and Generation

[0557] Server: Extracts pet appearance characteristics from photos and videos, and analyzes voice and behavioral patterns from videos and audio.

[0558] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[0559] 3. Preparing a Virtual Graveyard

[0560] Server: Allocates resources for the virtual cemetery and generates access links.

[0561] Terminal: Notify the user: "Your virtual graveyard is ready. Please access it at the following link: example_link."

[0562] 4. Dialogue and Interaction

[0563] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[0564] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[0565] Terminal: displays the response from the server and allows the user to continue the interaction.

[0566] The system allows users to virtually interact with deceased loved ones or pets, providing peace of mind.

[0567] The processing flow will be explained below.

[0568] Program processing steps

[0569] Step 1: Select and upload data

[0570] User: Launch the dedicated app and select photos, videos, audio, and text data of the deceased person or pet.

[0571] On the device: Preview the selected file and ask the user for confirmation. If the user presses the confirm button, the file will be uploaded to the server.

[0572] Server: Stores the received data in a database and notifies the device that the upload is complete.

[0573] Terminal: Notify the user that the upload was successful.

[0574] Step 2: Data analysis

[0575] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[0576] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[0577] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[0578] Server: Stores these feature data in a database.

[0579] Step 3: Creating a Virtual Being

[0580] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[0581] Server: Stores the generated virtual entities in a database for the virtual space.

[0582] Step 4: Preparing the Virtual Graveyard and Creating Links

[0583] Server: Allocates virtual graveyard resources and places the generated virtual beings in the virtual space.

[0584] Server: Generates a link to access the virtual cemetery and sends it to the device.

[0585] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[0586] Step 5: Access and Authentication

[0587] User: Clicks on the provided access link to access the virtual cemetery.

[0588] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[0589] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[0590] Step 6: Start a conversation

[0591] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[0592] Terminal: Sends user input to the server.

[0593] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[0594] Server: Sends the generated response to the terminal.

[0595] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[0596] Step 7: Additional Interactions

[0597] User: Continues to reminisce about and ask questions about deceased loved ones and pets. Performs specific actions in the virtual space, such as placing flowers.

[0598] Terminal: Sends user actions to the server and displays the results of the execution.

[0599] Server: Updates the virtual space in real time based on the user's actions, allowing the user to continue interacting.

[0600] In this way, users can virtually converse and interact with their deceased loved ones or pets, bringing them peace of mind.

[0601] Example 1

[0602] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0603] In modern society, people cherish memories of their deceased loved ones and pets and wish to feel close to them forever. However, existing technologies lack the means to recreate the presence of deceased people or pets in a virtual space and provide them in an interactive form using personal data related to them. Therefore, there is a need for a system that analyzes data related to deceased people or pets, generates virtual presences of the deceased or pets based on that data, and allows users to interact with them.

[0604] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0605] In this invention, the server includes: means for a user to select multiple pieces of personal data related to the deceased person or pet; means for previewing the selected personal data on a user terminal and confirming upload; means for transmitting the confirmed personal data to the server and saving it; means for analyzing the saved personal data and extracting characteristics of the deceased person or pet; means for generating a virtual being of the deceased person or pet using a generative AI model based on the extracted characteristic data; means for generating a 3D model of the generated virtual being and performing voice synthesis; means for storing the generated virtual being in a virtual space database and placing it in the virtual space; means for generating an access link to the virtual space and sending it to the user terminal; means for a user to access the virtual space and be authenticated by entering authentication information; means for enabling interaction with the virtual being in the virtual space if authentication is successful; means for analyzing messages from the interacting user and generating appropriate responses; and means for displaying the generated responses on the user terminal. This allows users to comprehensively organize data related to the deceased person or pet, generate virtual beings based on the data, and enjoy interacting with them.

[0606] "User" refers to a person who uses the system to manipulate personal data related to a deceased person or pet and to interact in a virtual space.

[0607] "Personal data" refers to digital information such as photos, videos, audio files, and text related to a deceased person or pet.

[0608] "User terminal" refers to a device used by a user to select, preview, review, and transmit personal data.

[0609] "Server" refers to a central computer system that receives, stores, and analyzes personal data sent by users and generates a virtual presence.

[0610] An "image analysis algorithm" refers to a computational method for extracting specific features from photos and videos.

[0611] "Speech analysis" refers to the process of extracting speech features from speech data.

[0612] "Text analytics" refers to the process of analyzing text data for frequently used words and speech patterns.

[0613] "Generative AI model" refers to an AI (artificial intelligence) model that uses machine learning technology to generate virtual versions of deceased people or pets.

[0614] "3D model generation" refers to the process of creating a three-dimensional virtual model based on characteristic data of a deceased person or pet.

[0615] "Speech synthesis" refers to the process of generating new speech from analyzed speech data.

[0616] "Database for virtual space" refers to a database for storing data on generated virtual entities.

[0617] "Access link" refers to a URL or the like that allows a user to access a virtual space.

[0618] "Authentication" refers to the process of verifying a user's identity when they access a system.

[0619] "Interaction interface" refers to a display and input system that allows a user and a virtual entity to interact in a virtual space.

[0620] "Prompt sentence" refers to an input sentence provided to an AI model to elicit an appropriate response.

[0621] "Response" refers to the appropriate response that a generative AI model gives in response to user input.

[0622] MODE FOR CARRYING OUT THE INVENTION

[0623] The present invention relates to a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system operates mainly between a server, a terminal, and users.

[0624] First, the user launches the app and selects personal data such as photos, videos, audio files, and text related to the deceased person or pet. The device previews the data and asks for confirmation of upload. Once confirmed, the data is sent from the user's device to the server using the HTTPS protocol. The server stores the received data and notifies the user that it has been successfully received.

[0625] The server then analyzes the stored data, which includes the following steps:

[0626] 1. Image analysis: Use OpenCV to extract facial landmark points and other features from photos and videos.

[0627] 2. Audio analysis: Using the LibROSA library, Mel-Frequency Cepstrum Coefficients (MFCCs) are extracted from the audio data and saved as feature data.

[0628] 3. Text Analysis: Use the NLTK library to tokenize text data and analyze it for frequently occurring words and phrases.

[0629] The server then uses a generative AI model (e.g., GPT-3) based on the extracted feature data to generate a virtual representation of the deceased person or pet, including:

[0630] 1. 3D model generation: Using Blender's API, a 3D model is automatically generated based on facial feature data.

[0631] 2. Voice synthesis: Voice data is generated based on the voice characteristics analyzed by Tacotron 2 and used for virtual characters.

[0632] The generated data is stored in a database for the virtual space, and the server allocates resources for the virtual cemetery and places the generated virtual beings in the virtual space. An access link to the virtual space is generated and sent to the user's terminal. The user's terminal notifies the user that the virtual cemetery is ready and provides the access link.

[0633] When a user clicks on the provided access link, they are given access to the virtual cemetery. Authentication is required for access, and the user enters their authentication information. Once access is authenticated, an interactive interface is displayed.

[0634] When a conversation begins, the server analyzes the message entered by the user and generates an appropriate response. For example, if the user asks, "How have you been?", a response that sounds like the character of the deceased person or pet will be returned (e.g., "I've been fine! How have you been?"). The response from the server is displayed on the user's device, allowing the user to continue the conversation.

[0635] Specific examples

[0636] Data collection

[0637] User: Use the dedicated app to select and upload photos and videos of your pet.

[0638] On your device: Displays "Uploading data" and notifies you "Upload complete" once it's finished.

[0639] Server: Stores the received data and prepares it for analysis.

[0640] Data analysis and generation

[0641] Server: Extracts pet appearance characteristics from photos and videos, and analyzes voice and behavioral patterns from videos and audio.

[0642] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[0643] Preparing for a virtual cemetery

[0644] Server: Allocates resources for the virtual cemetery and generates access links.

[0645] Terminal: Notify the user: "Your virtual graveyard is ready. Please access it at the following link: example_link."

[0646] Dialogue and Interaction

[0647] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[0648] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[0649] Terminal: displays the response from the server and allows the user to continue the interaction.

[0650] As described above, this invention allows users to comprehensively organize data related to deceased loved ones and pets, and based on that data, generate virtual beings and enjoy conversations with them. This system allows users to virtually converse with deceased loved ones and pets, providing peace of mind.

[0651] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0652] Step 1: The user launches the dedicated app and selects personal data (photos, videos, audio files, text) related to the deceased person or pet.

[0653] Input: The user operates the dedicated app and specifies personal data.

[0654] Specific operation: The user uses a smartphone or PC to select the relevant file from a folder using the "Data Selection" function within the app.

[0655] Step 2: Your device will preview the selected personal data and ask you to confirm the upload.

[0656] Input: A data file selected by the user.

[0657] Specific behavior: The device will display a preview window to allow the user to check the contents of the data, and a confirmation dialog will appear with the message "Do you want to upload?"

[0658] Step 3: The user confirms the upload and the device sends the data to the server.

[0659] Input: User upload confirmation.

[0660] Output: The data sent to the server.

[0661] What it does: When the user presses the "Upload" button, the device compresses the selected data and sends it to the server using the HTTPS protocol.

[0662] Step 4: The server stores the received data and notifies the user of successful reception.

[0663] Input: Data from the terminal.

[0664] Output: Saved data and notification to the user.

[0665] Specific operation: The server stores the received data in a database and returns an HTTP response saying "Data upload completed." The device receives this response and notifies the user.

[0666] Step 5: The server analyzes the received personal data.

[0667] Input: Stored personal data.

[0668] Output: Extracted feature data.

[0669] Specific behavior:

[0670] Image analysis: Extract facial landmark points from photos and videos using OpenCV.

[0671] Audio analysis: Extract Mel-Frequency Cepstral Coefficients (MFCCs) from audio files using the LibROSA library.

[0672] Text analysis: Use the NLTK library to tokenize text data and analyze frequently occurring words and phrases.

[0673] Step 6: The server uses the extracted feature data to generate a virtual presence of the deceased person or pet using a generative AI model.

[0674] Input: Extracted feature data.

[0675] Output: Data of the generated virtual entity.

[0676] Specific behavior:

[0677] A generative AI model (e.g., GPT-3) is trained on collected text patterns to generate hypothetical conversational responses.

[0678] Using Blender's API, a 3D model is generated based on facial feature data.

[0679] Voice data is generated based on the voice characteristics analyzed by Tacotron 2.

[0680] Step 7: The server stores the generated data in a database for the virtual space and places it in the virtual space.

[0681] Input: Data of the generated virtual entity.

[0682] Output: A virtual entity placed in virtual space.

[0683] How it works: The server uses the Unity server to assign virtual characters to virtual spaces and save the corresponding metadata in a database.

[0684] Step 8: The server generates an access link to the virtual space and sends it to the terminal.

[0685] Input: Virtual space resource information.

[0686] Output: Access link.

[0687] Specific operation: The server returns a dynamically generated URL link to the device in an HTTP response, and the device provides the link to the user.

[0688] Step 9: The user clicks on the provided access link to access the virtual cemetery, enters their authentication information and is authenticated.

[0689] Input: Access link, authentication information.

[0690] Output: Authenticated access to the virtual space.

[0691] Specific operation: The user opens the access link in a browser and enters their user ID and password on the displayed login screen. The server performs authentication, and if successful, they enter the virtual space.

[0692] Step 10: The user inputs a message into the dialogue interface in the virtual space, and the server generates an appropriate response.

[0693] Input: The user's message.

[0694] Output: The appropriate response.

[0695] Specific operation: The server receives the user's message, sends a prompt such as "The user is asking how are you?" to the GPT-3 model, and receives the generated response.

[0696] Step 11: The server sends the generated response to the user terminal and displays it to the user.

[0697] Input: The server-generated response.

[0698] Output: The response displayed on the user's terminal.

[0699] Specific operation: The server sends the generated response as an HTTP response to the user's terminal, and the terminal displays it to the user in text or audio format.

[0700] (Application example 1)

[0701] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0702] Previous virtual presence systems related to deceased loved ones or pets lacked the interactive services that users could experience directly in a physical store. This prevented users from feeling closer to their loved ones or pets through virtual interactions, making it difficult to fully experience peace of mind. Furthermore, without incorporating advanced AI models to generate natural interactions, the user experience was unnatural and did not feel as close to real interactions.

[0703] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0704] In this invention, the server includes means for acquiring a plurality of personal data related to the deceased person or pet, means for analyzing the acquired personal data and extracting characteristics of the deceased person or pet, means for generating a virtual presence of the deceased person or pet based on the extracted characteristics, means for providing an interaction interface that allows a user to interact with the virtual presence in a physical store, and means for using a generative AI model to generate natural interactions when the user interacts with the virtual presence, thereby enabling users to interact with the deceased person or pet in a natural way in a physical store and gain peace of mind.

[0705] "Personal data" refers to information such as photos, videos, audio files, and text related to an individual.

[0706] "Features" refer to attributes of the deceased person or pet, such as their appearance, voice, or language usage patterns, extracted from image, audio, and text data.

[0707] "Virtual presence" refers to a virtual character that can be interacted with, including a 3D model and voice, generated from the personal data of a deceased person or pet.

[0708] "Interaction interface" refers to a device, such as a tablet, robot, or smart glasses, that a user uses to communicate with a virtual being.

[0709] A "generative AI model" refers to an artificial intelligence model that generates natural responses to user input.

[0710] "Brick and mortar store" refers to a physical store that a user can actually visit.

[0711] The present invention provides a system that allows users to experience virtual interactions with deceased loved ones or pets in a physical store. This system provides a comprehensive service, from collecting personal data to generating virtual beings and interacting with them through an interface. Specific embodiments are described below.

[0712] Data collection

[0713] Users use dedicated terminals (tablets, robots, or smart glasses) installed in physical stores to input personal data related to the deceased person or pet. This personal data includes photos, videos, audio files, and text. The dedicated terminals then send this data to a server.

[0714] Data analysis and virtual entity generation

[0715] The server analyzes the received personal data. Image data is extracted using DeepFace to identify physical features of the deceased person or pet, while audio data is extracted using Google Text-to-Speech (gTTS) and other voice analysis tools. Text data is analyzed using the OpenAI API to identify frequently used words and speech patterns.

[0716] Based on this feature data, a generative AI model is used to generate a virtual entity, which includes a 3D model and a synthesized voice, and is placed in a virtual space.

[0717] Providing a conversational interface

[0718] The dedicated terminal provides the user with a dialogue interface through which the user can interact with the virtual entity. Questions and comments entered by the user are sent to the server.

[0719] The server analyzes the user's input using a generative AI model and generates an appropriate response. The response is designed to be natural and takes into account the characteristics of the deceased person or pet. The generated response is sent to a dedicated terminal and displayed to the user.

[0720] Specific examples

[0721] For example, if a user uploads photos and videos of their beloved dog "Pochi" in a physical store and asks, "Pochi, how have you been lately?", the server analyzes this data and generates a virtual Pochi. Using a generative AI model, it generates a response to the question, "Pochi, how have you been lately?", returning the response, "I've been fine! How have you been?"

[0722] Prompt Sentence Examples

[0723] Below are some examples of prompt sentences.

[0724] Please reply as Pochi to the following questions: What have you been up to lately?

[0725] The system thus configured allows users to interact with virtual deceased loved ones or pets through natural conversations within a physical store, providing peace of mind.

[0726] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0727] Step 1:

[0728] Users use dedicated terminals (tablets, robots, or smart glasses) installed in physical stores to input personal data such as photos, videos, audio files, and text related to the deceased person or pet. The input data is sent to the server by the terminal. The input is completed when the user selects the data and presses the "upload" button.

[0729] Step 2:

[0730] The server stores the personal data received from the device and then begins analyzing it. Specifically, image data from photos and videos is used to extract physical features of the deceased person or pet using DeepFace. Voice data is used to extract features using Google Text-to-Speech (gTTS) and other voice analysis tools. Text data is analyzed using the OpenAI API to identify frequently used words and speaking patterns.

[0731] Step 3:

[0732] The server uses a generative AI model to generate a virtual presence (3D model and voice) based on the feature data extracted in step 2. This virtual presence reflects the unique characteristics of the deceased person or pet and is capable of natural conversation. The generated virtual presence is stored in a database for the virtual space.

[0733] Step 4:

[0734] The dedicated terminal provides a dialogue interface to the user. The user can use this interface to start a dialogue with the virtual entity. For example, when the user types "Pochi, how have you been lately?", the data is sent from the terminal to the server.

[0735] Step 5:

[0736] The server analyzes the user's questions and comments received in step 4 and generates an appropriate response using the generative AI model. This response is designed to be a natural conversation that takes into account the characteristics of the deceased person or pet. For example, in response to a user's question, "Pochi, how have you been lately?", the server generates a response such as, "I've been fine! How have you been?"

[0737] Step 6:

[0738] The generated response is sent from the server to a dedicated terminal and displayed to the user. The user can confirm the response and continue the dialogue. The content of the dialogue is recorded on the server and used as reference for future dialogues.

[0739] Step 7:

[0740] When the user finishes the conversation, the dedicated terminal closes the conversation interface and the server ends the session. If the user wishes, the conversation content can be saved and a link for the next access can be provided.

[0741] Through the above steps, users can have natural conversations with their deceased loved ones or pets in a physical store, providing peace of mind.

[0742] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0743] The present invention is a system that allows users to create virtual beings based on personal data related to deceased people or pets and engage in conversations in a virtual space. This system incorporates an emotion engine that analyzes the user's emotions, enabling more personalized conversations. The system operates primarily between a server, a terminal, and the user.

[0744] User Data Collection

[0745] 1. Data Collection

[0746] User: The user launches the app and selects personal data related to the deceased person or pet, including photos, videos, audio files, and text.

[0747] On the device: The device will preview the data you selected and ask you to confirm the upload. Once confirmed, the data will be uploaded to the server.

[0748] Server: The server stores the received data and notifies the user that it was received successfully.

[0749] Data analysis and generation

[0750] 2. Data Analysis

[0751] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[0752] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[0753] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[0754] Server: Stores these feature data in a database.

[0755] 3. Virtual Being Creation

[0756] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[0757] Server: Stores the generated virtual entities in a database for the virtual space.

[0758] Preparing for a virtual cemetery

[0759] 4. Resource Allocation and Link Creation

[0760] Server: Allocates resources for the virtual graveyard and places the generated virtual beings in the virtual space.

[0761] Server: Generates a link to access the virtual cemetery and sends it to the device.

[0762] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[0763] Use of dialogue and emotion engines

[0764] 5. Access and Authentication

[0765] User: Clicks on the provided access link to access the virtual cemetery.

[0766] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[0767] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[0768] 6. Start a dialogue

[0769] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[0770] Terminal: Sends user input to the server.

[0771] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[0772] Server: Sends the generated response to the terminal.

[0773] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[0774] Use of emotion engine

[0775] 7. Emotional Recognition

[0776] Device: The device is equipped with facial recognition and voice analysis functions, and analyzes the user's emotions in real time from their facial expressions and voice.

[0777] Server: Receives the analysis results and sends them to the emotion engine.

[0778] Server: The emotion engine adjusts the content and tone of the response based on the user's emotional data.

[0779] 8. Personalized responses

[0780] Server: Generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it will return words of encouragement.

[0781] Server: Sends the generated response to the terminal.

[0782] Terminal: Displays the response received from the server to the user, for example, "Cheer up, we're always thinking about you."

[0783] Specific examples

[0784] As an example, we will explain how a user can upload photos or videos of a deceased pet and start a dialogue using the emotion engine.

[0785] 1. Data Collection

[0786] User: Use the dedicated app to select and upload photos and videos of your pet.

[0787] On your device: Display the progress of the data upload and notify you when it's finished by saying "Upload complete."

[0788] Server: Stores the received data and prepares it for analysis.

[0789] 2. Data Analysis and Generation

[0790] Server: Extracts the pet's appearance characteristics from photos and videos and analyzes its voice characteristics.

[0791] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[0792] 3. Preparing a Virtual Graveyard

[0793] Server: Allocates resources for the virtual cemetery and generates access links.

[0794] Terminal: Send the link to the user and tell them, "Your virtual cemetery is ready. Please access it from the following link: example_link."

[0795] 4. Use of dialogue and emotion engines

[0796] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[0797] Terminal: Sends user input to the server.

[0798] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[0799] Terminal: Displays the response from the server and shows it to the user.

[0800] Terminal: Analyzes emotions from the user's facial expressions and voice, and sends the results to the server.

[0801] Server: The emotion engine generates a response based on the user's emotions, for example, returning encouraging words such as "You're sad, we're thinking of you too."

[0802] The system allows users to virtually interact with deceased loved ones or pets and receive personalized responses based on their emotions, providing greater peace of mind.

[0803] The processing flow will be explained below.

[0804] Program processing steps

[0805] Step 1: Select and upload data

[0806] User: Launch the dedicated app and select photos, videos, audio, and text data of the deceased person or pet.

[0807] On the device: Preview the selected file and ask the user for confirmation. If the user presses the confirm button, the file will be uploaded to the server.

[0808] Server: Stores the received data in a database and notifies the device that the upload is complete.

[0809] Terminal: Notify the user that the upload was successful.

[0810] Step 2: Data analysis

[0811] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[0812] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[0813] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[0814] Server: Stores these feature data in a database.

[0815] Step 3: Creating a Virtual Being

[0816] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[0817] Server: Stores the generated virtual entities in a database for the virtual space.

[0818] Step 4: Preparing the Virtual Graveyard and Creating Links

[0819] Server: Allocates resources for the virtual graveyard and places the generated virtual beings in the virtual space.

[0820] Server: Generates a link to access the virtual cemetery and sends it to the device.

[0821] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[0822] Step 5: Access and Authentication

[0823] User: Clicks on the provided access link to access the virtual cemetery.

[0824] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[0825] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[0826] Step 6: Start a conversation

[0827] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[0828] Terminal: Sends user input to the server.

[0829] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[0830] Server: Sends the generated response to the terminal.

[0831] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[0832] Step 7: Recognize emotions

[0833] Device: The device is equipped with facial recognition and voice analysis functions, and analyzes the user's emotions in real time from their facial expressions and voice.

[0834] Server: Receives the analysis results and sends them to the emotion engine.

[0835] Server: The emotion engine adjusts the content and tone of the response based on the user's emotional data.

[0836] Step 8: Personalize your response

[0837] Server: Generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it will return words of encouragement.

[0838] Server: Sends the generated response to the terminal.

[0839] Terminal: Displays the response received from the server to the user, for example, "Cheer up, we're always thinking about you."

[0840] Step 9: Additional Interactions

[0841] User: Continues to reminisce about and ask questions about deceased loved ones and pets. Performs specific actions in the virtual space, such as placing flowers.

[0842] Terminal: Sends user actions to the server and displays the results of the execution.

[0843] Server: Updates the virtual space in real time based on the user's actions, allowing the user to continue interacting.

[0844] In this way, users can virtually converse and interact with their deceased loved ones or pets, bringing them peace of mind.

[0845] Example 2

[0846] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0847] Today, many people use photos and videos to remember their deceased loved ones and pets, but these alone are not enough to bring them peace of mind. While there are systems that allow users to virtually interact with their deceased loved ones and pets, few systems can respond to emotional changes in real time. As a result, it is currently difficult for users to receive personalized experiences tailored to their emotions.

[0848] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to select multiple pieces of personal data related to the deceased person or pet; means for transmitting the selected personal data to the server; means for analyzing the personal data received by the server and extracting characteristics of the deceased person or pet; means for generating a virtual presence of the deceased person or pet using a generative AI model based on the extracted characteristics; means for placing the generated virtual presence in a virtual space; means for generating and providing a link for accessing the virtual space to the user; means for the user to interact with the virtual presence in the virtual space; an emotion engine for analyzing the user's emotions in real time; and means for personalizing responses based on the emotion engine. This allows the user to receive personalized responses according to their emotions while virtually interacting with the deceased person or pet.

[0849] "User" refers to an individual who uses this system.

[0850] "Personal data" refers to information such as photos, videos, audio files, and text related to the deceased person or pet.

[0851] "Terminal" refers to a device operated by a user, such as a computer, smartphone, or tablet.

[0852] "Server" refers to the computer that processes and stores data at the center of this system.

[0853] "Analysis" refers to the process of extracting characteristics and information about the deceased person or pet from personal data.

[0854] A "generative AI model" refers to an artificial intelligence model that uses machine learning algorithms to generate new data or formats.

[0855] "Virtual presence" refers to a digital representation of a deceased person or pet created by a generative AI model based on analyzed personal data.

[0856] "Virtual space" refers to a digital environment in which users interact with virtual beings.

[0857] "Link" refers to a URL or hyperlink for accessing a virtual space.

[0858] An "emotion engine" refers to a system that analyzes a user's emotions and generates a response based on those emotions.

[0859] The present invention provides a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system is mainly composed of a server, terminals, and users.

[0860] User Data Collection

[0861] User: Launches the dedicated app and selects photos, videos, audio files, and text files related to the deceased person or pet.

[0862] On the device: Preview the selected data and ask the user to confirm the upload. Once confirmed, send the data to the server.

[0863] Server: Saves the received data in local storage or cloud storage and notifies the user that the upload is complete.

[0864] Data analysis and generation

[0865] Server: Analyzes the received photos and videos using image analysis algorithms (e.g., OpenCV or TensorFlow) to extract the external characteristics of the deceased person or pet.

[0866] Server: The audio file is analyzed using a speech analysis algorithm (e.g., Google Cloud Speech-to-Text or IBM Watson) to extract voice characteristics.

[0867] Server: Analyzes the text data using natural language processing algorithms (e.g., spaCy or BERT) to extract frequently used words and distinctive language patterns.

[0868] Server: Stores these feature data in a database (e.g., PostgreSQL or MongoDB).

[0869] Server: Based on the stored feature data, a generative AI model (e.g., GPT-3 or StyleGAN) is used to generate virtual deceased people or pets. This generation process includes creating 3D models and synthesizing voices. The generated virtual beings are stored in a database for the virtual space.

[0870] Preparing for a virtual cemetery

[0871] Server: Allocate resources for the virtual cemetery and place the virtual presence of deceased people and pets in the virtual space. This includes securing cloud server resources and generating the virtual space.

[0872] Server: Generates and provides users with a dedicated link to access the virtual cemetery.

[0873] Terminal: Notifies the user of the generated access link and informs them that the virtual cemetery is ready.

[0874] Use of dialogue and emotion engines

[0875] User: Clicks on the provided access link to access the virtual cemetery.

[0876] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[0877] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[0878] User: Initiate a virtual conversation with a deceased loved one or pet, for example, by typing "How have you been?"

[0879] Terminal: Sends user input to the server.

[0880] Server: Analyzes the user's input using an AI model, generates responses that the deceased person or pet would likely make, and sends them to the device.

[0881] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[0882] Use of emotion engine

[0883] Device: Using facial recognition and voice analysis functions, emotions are analyzed in real time from the user's facial expressions and voice.

[0884] Server: Receives the analysis results and sends them to the emotion engine.

[0885] Server: The emotion engine generates personalized responses by adjusting the content and tone of the response based on the user's emotional data. For example, if the user is sad, it adds words of encouragement.

[0886] Server: Sends the generated response to the terminal and displays it to the user, for example, "Cheer up, I'm always worried about you."

[0887] Specific examples

[0888] For example, consider a user uploading photos and videos of a deceased pet to use the system:

[0889] 1. User: Use the dedicated app to select and upload photos and videos of your pet.

[0890] 2. Device: Displays the progress of data upload and notifies you when it is complete.

[0891] 3. Server: Stores the received data and prepares it for analysis.

[0892] 4. Server: Extracts the pet's external features from photos and videos and analyzes its audio features.

[0893] 5. Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[0894] 6. Server: Allocates resources for the virtual cemetery and generates access links.

[0895] 7. Terminal: Notify the user of the link and say, "Your virtual cemetery is ready. Please access it from the following link: example_link."

[0896] 8. User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[0897] 9. Terminal: Sends user input to the server and displays responses to the user. It displays responses such as "Pochi was fine! How was yours?"

[0898] 10. Terminal: Analyzes emotions from the user's facial expressions and voice and sends them to the server.

[0899] 11. Server: The emotion engine generates a response based on the user's emotions, returning encouraging words such as "You're sad, we're thinking of you too." This system allows the user to find greater peace of mind.

[0900] This Detailed Description provides a detailed understanding of how the present invention may be practiced.

[0901] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0902] Program processing steps

[0903] Step 1: Collect user data

[0904] Input: The user selects photos, videos, audio files, and text related to the deceased person or pet.

[0905] How it works: Launch the dedicated app and select data through the interface.

[0906] Output: The selected data is saved to the device and sent to the next processing step.

[0907] Step 2: Upload data

[0908] Input: Personal data stored on your device.

[0909] How it works: The device previews the selected data and asks the user to confirm the upload. Once confirmed, the data is sent to the server.

[0910] Output: Personal data is uploaded to the server.

[0911] Step 3: Data storage and notification

[0912] Input: Personal data received by the server.

[0913] How it works: The server saves the received data to local storage or cloud storage and notifies the user that the upload is complete.

[0914] Output: The data is saved and the user is notified that the upload is complete.

[0915] Step 4: Image analysis

[0916] Input: Photos and videos from your saved personal data.

[0917] How it works: The server uses image analysis algorithms (e.g., OpenCV or TensorFlow) to extract the external features of the deceased person or pet.

[0918] Output: Appearance feature data is generated.

[0919] Step 5: Audio analysis

[0920] Input: Audio file from saved personal data.

[0921] How it works: The server uses voice analysis algorithms (e.g., Google Cloud Speech-to-Text or IBM Watson) to extract voice characteristics and tone.

[0922] Output: Voice feature data is generated.

[0923] Step 6: Text Analysis

[0924] Input: A text file from your saved personal data.

[0925] How it works: The server uses natural language processing algorithms (e.g. spaCy or BERT) to analyze frequently used words and unique language patterns.

[0926] Output: Word and language pattern data is generated.

[0927] Step 7: Save to database

[0928] Input: Analyzed appearance feature data, voice feature data, and word and language pattern data.

[0929] How it works: The server stores this data in a database (e.g. PostgreSQL or MongoDB).

[0930] Output: Feature data stored in a database.

[0931] Step 8: Virtual Being Generation

[0932] Input: Stored feature data.

[0933] How it works: The server uses generative AI models (e.g., GPT-3 or StyleGAN) to generate virtual versions of deceased people and pets, including creating 3D models and synthesizing voices.

[0934] Output: Virtual entity data is generated.

[0935] Step 9: Virtual Space Placement

[0936] Input: Data of the generated virtual being.

[0937] How it works: The server allocates resources for the virtual cemetery and places virtual figures of deceased people and pets in the virtual space.

[0938] Output: A virtual presence of a deceased person or pet placed in a virtual space.

[0939] Step 10: Link generation and notification

[0940] Input: A virtual entity placed in a virtual space.

[0941] How it works: The server generates a dedicated link to access the virtual cemetery and provides it to the user.

[0942] Output: The generated access link is notified to the user.

[0943] Step 11: Authentication and Access

[0944] Input: The access link the user received.

[0945] How it works: The user clicks on the provided link to access the virtual cemetery. The device sends authentication information to the server, which then performs the authentication process.

[0946] Output: The user can now access the virtual graveyard.

[0947] Step 12: Start a conversation

[0948] Input: A prompt that the user types in the virtual space (e.g., "How have you been?").

[0949] How it works: The device sends user input to the server, which then analyzes the prompt and uses a generative AI model to generate a response for the deceased person or pet. The generated response is then sent to the device.

[0950] Output: The user will see a response from the deceased person or pet (e.g., "I'm fine! How have you been?").

[0951] Step 13: Emotion Recognition

[0952] Input: The user's facial expressions and voice during the conversation.

[0953] How it works: The device uses facial recognition and voice analysis to analyze the user's facial expressions and voice in real time.

[0954] Output: The parsed emotion data is generated.

[0955] Step 14: Personalize your response

[0956] Input: Parsed emotion data and the following user input:

[0957] Operation: The server uses the emotion engine to generate a response based on the user's emotion. The generated response is sent to the device.

[0958] Output: A personalized response is displayed to the user (e.g., "Cheer up, we're always worried about you").

[0959] This allows the user to enjoy a virtual conversation while receiving individual responses according to their emotions.

[0960] (Application example 2)

[0961] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0962] Conventional virtual dialogue systems have the problem that interactions with virtual beings generated based on personal data related to deceased loved ones or pets cannot flexibly respond to the user's emotions or circumstances, which limits the peace of mind that users can gain and makes it difficult to achieve personalized dialogue.

[0963] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for importing multiple pieces of personal data related to the deceased person or pet, means for analyzing the imported personal data and extracting characteristics of the deceased person or pet, means for generating a virtual presence of the deceased person or pet based on the extracted characteristics, means for providing a virtual space in which the user can interact with the generated virtual presence, means including an emotion engine for analyzing the user's emotions, and means for adjusting the content and tone of the response based on the analyzed emotion data. This enables personalized dialogue according to the user's emotions and situation.

[0964] "Deceased person or pet" refers to an object that the user recognizes as a deceased person or pet and recreates in the virtual space.

[0965] "Personal Data" refers to data containing personal information, such as photographs, videos, audio files, and text related to a deceased person or pet.

[0966] "Means for extracting features" refers to the processes and techniques for analyzing and extracting the appearance, vocal characteristics, and language patterns of the deceased person or pet from the collected personal data.

[0967] "Means for generating virtual entities" refers to the technology and process for generating virtual entities, such as 3D models and voices of deceased individuals or pets, based on extracted feature data.

[0968] "Virtual space" refers to a digitally simulated environment in which users can interact with generated virtual beings.

[0969] An "emotion engine" refers to software and algorithms that analyze a user's emotions from their facial expressions and voice and adjust the content and tone of their responses accordingly.

[0970] "Means for providing an interactive virtual space" refers to the technologies and processes for constructing and making accessible a virtual space that allows users to interact with generated virtual beings.

[0971] "Emotion data" refers to information about emotions obtained by analyzing the user's facial expressions and voice.

[0972] "Response content" refers to the content of a reply to an input made by a user to a virtual entity.

[0973] "Tone of response" refers to the nuances and tone of voice of the response to the user that are adjusted by the emotion engine.

[0974] An embodiment of the present invention will be described in detail below. The present invention is a system that allows a user to generate a virtual entity based on personal data related to a deceased person or pet and to engage in a dialogue in a virtual space. The system incorporates an emotion engine and includes a function for analyzing the user's emotions and providing a more personalized dialogue.

[0975] Users use a dedicated application to collect personal data related to the deceased person or pet. This application is installed on a smartphone and allows users to upload data such as photos, videos, audio files, and text. The device previews the data selected by the user, asks for confirmation of the upload, and once confirmation is received, sends the data to the server. The server stores the received data and notifies the user that it has been successfully received.

[0976] Using the generative AI model, the server analyzes the received data and extracts the characteristics of the deceased person or pet. Specifically, it uses an image analysis algorithm to extract external features from photos and videos, and an audio analysis algorithm to extract voice characteristics and tone from audio data. It also analyzes frequently used words and unique language patterns from text data and stores this feature data in a database. Based on the analyzed feature data, the server creates a 3D model and synthesizes voice to generate a virtual deceased person or pet.

[0977] The generated virtual beings are placed in a virtual space. The server allocates resources for the virtual space, places the generated virtual beings in the virtual space, and generates a link to access the virtual cemetery. This link is provided to the user via the terminal, and the user clicks on the provided link to access the virtual cemetery. The terminal sends the user's authentication information to the server, and if the authentication is successful, a dialogue system is launched and guides the user to the virtual cemetery.

[0978] Users can interact with their deceased loved ones or pets in a virtual space. For example, by inputting a question such as "How have you been?", the device sends the input to the server, which then analyzes it using a generative AI model to generate a response that the deceased or pet would likely give. The generated response is then sent to the device and displayed to the user.

[0979] The device also provides a function that analyzes the user's emotions in real time using an emotion engine. The device is equipped with facial recognition and voice analysis functions, which analyze the user's emotions from their facial expressions and voice and send the results to the server. The server then uses the emotion engine to adjust the content and tone of the response based on the analysis results, generating a personalized response that matches the user's emotions.

[0980] Here are some examples of prompts:

[0981] User: Pochi, what have you been up to lately?

[0982] System: Pochi was fine! How were you?

[0983] The system allows users to virtually interact with deceased loved ones or pets and receive personalized responses based on their emotions, providing greater peace of mind.

[0984] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0985] Step 1:

[0986] The user launches a dedicated application and selects personal data (photos, videos, audio files, text, etc.) related to the deceased person or pet. The application prompts the user to upload the selected data. The application previews the data selected by the user and asks for confirmation of the upload. Once the user confirms, the device sends the data to the server. The input is the personal data selected by the user, and the output is the transmission of the data to the server.

[0987] Step 2:

[0988] The server stores the received personal data and notifies the user that it has been received successfully. The server then uses image analysis algorithms to analyze the photos and videos to extract the physical characteristics of the deceased person or pet. It also uses voice analysis algorithms to analyze the audio data to extract voice characteristics and tone. It also extracts frequently used words and distinctive language patterns from the text data. The input is the personal data uploaded to the server, and the output is data on physical characteristics, voice characteristics, and language patterns.

[0989] Step 3:

[0990] The server uses a generative AI model to generate a virtual deceased person or pet based on the analyzed feature data. This generation process includes creating a 3D model and synthesizing voice. The data of the generated virtual being is stored in a database for the virtual space. The input is the extracted feature data, and the output is the data of the generated virtual being.

[0991] Step 4:

[0992] The server allocates resources for the virtual cemetery and places the generated virtual beings in the virtual space. Next, it generates a link to access the virtual cemetery and sends it to the terminal. The terminal notifies the user of this link and informs them that the virtual cemetery is ready. The input is the data of the virtual beings, and the output is the generation and notification of the access link.

[0993] Step 5:

[0994] The user clicks on the provided access link to access the virtual cemetery. The terminal sends the user's authentication information to the server and executes the authentication process. If the authentication is successful, the server launches a dialogue system and guides the user to the virtual cemetery. The input is the user's access link and authentication information, and the output is access to the virtual cemetery and the launch of the dialogue system.

[0995] Step 6:

[0996] The user interacts with the deceased person or pet in the virtual space. For example, they might type, "How have you been?" The device sends the user's input to the server. The server analyzes the user's input using a generative AI model and generates a response that the deceased person or pet would likely give. The generated response is sent to the device and displayed to the user. The input is the user's interaction, and the output is the generated response.

[0997] Step 7:

[0998] The device uses facial recognition and voice analysis functions to analyze emotions from the user's facial expressions and voice. The analysis results are sent to the server. The server's emotion engine adjusts the response content and tone based on the user's emotional data. The input is the user's facial expressions and voice data, and the output is the adjusted response content and tone.

[0999] Step 8:

[1000] The server generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it generates words of encouragement and sends the generated response to the device, which then displays it to the user. The input is the analysis results of the emotion engine, and the output is a personalized response.

[1001] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1002] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1003] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1004] [Third embodiment]

[1005] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1006] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1007] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1008] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1009] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1010] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1011] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1012] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1013] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1014] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1015] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1016] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1017] The present invention provides a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system operates mainly between a server, a terminal, and users.

[1018] User Data Collection

[1019] 1. Data Collection

[1020] User: The user launches the app and selects personal data related to the deceased person or pet, including photos, videos, audio files, and text.

[1021] On your device: The device will preview the data you selected and ask you to confirm the upload. Once confirmed, the data will be sent to the server.

[1022] Server: The server stores the received data and notifies the user that it was received successfully.

[1023] Data analysis and generation

[1024] 2. Data Analysis

[1025] Server: The server analyzes the received personal data. It uses image analysis algorithms to extract features of the deceased person or pet from photos and videos, analyzes audio data to extract voice characteristics, and analyzes text data to identify frequently used words and speaking patterns.

[1026] 3. Virtual Being Creation

[1027] Server: Based on the extracted feature data, a generative AI model is used to generate a virtual presence of the deceased person or pet, including 3D model generation and voice synthesis.

[1028] Server: Stores the generated data in a database for the virtual space.

[1029] Preparing for a virtual cemetery

[1030] 4. Resource Allocation and Link Creation

[1031] Server: Allocates resources for the virtual graveyard, places the generated virtual beings in the virtual space, generates access links to the virtual space, and sends them to the terminals.

[1032] Terminal: The terminal notifies the user that the virtual cemetery is ready and provides an access link.

[1033] Dialogue and Interaction

[1034] 5. Access and Interaction

[1035] User: The user clicks on the provided access link to access the virtual cemetery. Authentication is required for access, ensuring security.

[1036] Terminal: Once access to the virtual graveyard is authenticated, an interactive interface is displayed.

[1037] 6. Start a dialogue

[1038] Server: Analyzes the message entered by the user and generates an appropriate response. For example, the question "How have you been?" will be answered by a character that resembles a deceased person or pet.

[1039] Terminal: Receives the response from the server and displays it to the user.

[1040] Specific examples

[1041] As an example, we will explain the process when a user uploads photos and videos of a deceased pet.

[1042] 1. Data Collection

[1043] User: Use the dedicated app to select and upload photos and videos of your pet.

[1044] On your device: Displays "Uploading data" and notifies you "Upload complete" once it's finished.

[1045] Server: Stores the received data and prepares it for analysis.

[1046] 2. Data Analysis and Generation

[1047] Server: Extracts pet appearance characteristics from photos and videos, and analyzes voice and behavioral patterns from videos and audio.

[1048] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[1049] 3. Preparing a Virtual Graveyard

[1050] Server: Allocates resources for the virtual cemetery and generates access links.

[1051] Terminal: Notify the user: "Your virtual graveyard is ready. Please access it at the following link: example_link."

[1052] 4. Dialogue and Interaction

[1053] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[1054] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[1055] Terminal: displays the response from the server and allows the user to continue the interaction.

[1056] The system allows users to virtually interact with deceased loved ones or pets, providing peace of mind.

[1057] The processing flow will be explained below.

[1058] Program processing steps

[1059] Step 1: Select and upload data

[1060] User: Launch the dedicated app and select photos, videos, audio, and text data of the deceased person or pet.

[1061] On the device: Preview the selected file and ask the user for confirmation. If the user presses the confirm button, the file will be uploaded to the server.

[1062] Server: Stores the received data in a database and notifies the device that the upload is complete.

[1063] Terminal: Notify the user that the upload was successful.

[1064] Step 2: Data analysis

[1065] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[1066] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[1067] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[1068] Server: Stores these feature data in a database.

[1069] Step 3: Creating a Virtual Being

[1070] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[1071] Server: Stores the generated virtual entities in a database for the virtual space.

[1072] Step 4: Preparing the Virtual Graveyard and Creating Links

[1073] Server: Allocates virtual graveyard resources and places the generated virtual beings in the virtual space.

[1074] Server: Generates a link to access the virtual cemetery and sends it to the device.

[1075] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[1076] Step 5: Access and Authentication

[1077] User: Clicks on the provided access link to access the virtual cemetery.

[1078] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[1079] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[1080] Step 6: Start a conversation

[1081] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[1082] Terminal: Sends user input to the server.

[1083] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[1084] Server: Sends the generated response to the terminal.

[1085] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[1086] Step 7: Additional Interactions

[1087] User: Continues to reminisce about and ask questions about deceased loved ones and pets. Performs specific actions in the virtual space, such as placing flowers.

[1088] Terminal: Sends user actions to the server and displays the results of the execution.

[1089] Server: Updates the virtual space in real time based on the user's actions, allowing the user to continue interacting.

[1090] In this way, users can virtually converse and interact with their deceased loved ones or pets, bringing them peace of mind.

[1091] Example 1

[1092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1093] In modern society, people cherish memories of their deceased loved ones and pets and wish to feel close to them forever. However, existing technologies lack the means to recreate the presence of deceased people or pets in a virtual space and provide them in an interactive form using personal data related to them. Therefore, there is a need for a system that analyzes data related to deceased people or pets, generates virtual presences of the deceased or pets based on that data, and allows users to interact with them.

[1094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1095] In this invention, the server includes: means for a user to select multiple pieces of personal data related to the deceased person or pet; means for previewing the selected personal data on a user terminal and confirming upload; means for transmitting the confirmed personal data to the server and saving it; means for analyzing the saved personal data and extracting characteristics of the deceased person or pet; means for generating a virtual being of the deceased person or pet using a generative AI model based on the extracted characteristic data; means for generating a 3D model of the generated virtual being and performing voice synthesis; means for storing the generated virtual being in a virtual space database and placing it in the virtual space; means for generating an access link to the virtual space and sending it to the user terminal; means for a user to access the virtual space and be authenticated by entering authentication information; means for enabling interaction with the virtual being in the virtual space if authentication is successful; means for analyzing messages from the interacting user and generating appropriate responses; and means for displaying the generated responses on the user terminal. This allows users to comprehensively organize data related to the deceased person or pet, generate virtual beings based on the data, and enjoy interacting with them.

[1096] "User" refers to a person who uses the system to manipulate personal data related to a deceased person or pet and to interact in a virtual space.

[1097] "Personal data" refers to digital information such as photos, videos, audio files, and text related to a deceased person or pet.

[1098] "User terminal" refers to a device used by a user to select, preview, review, and transmit personal data.

[1099] "Server" refers to a central computer system that receives, stores, and analyzes personal data sent by users and generates a virtual presence.

[1100] An "image analysis algorithm" refers to a computational method for extracting specific features from photos and videos.

[1101] "Speech analysis" refers to the process of extracting speech features from speech data.

[1102] "Text analytics" refers to the process of analyzing text data for frequently used words and speech patterns.

[1103] "Generative AI model" refers to an AI (artificial intelligence) model that uses machine learning technology to generate virtual versions of deceased people or pets.

[1104] "3D model generation" refers to the process of creating a three-dimensional virtual model based on characteristic data of a deceased person or pet.

[1105] "Speech synthesis" refers to the process of generating new speech from analyzed speech data.

[1106] "Database for virtual space" refers to a database for storing data on generated virtual entities.

[1107] "Access link" refers to a URL or the like that allows a user to access a virtual space.

[1108] "Authentication" refers to the process of verifying a user's identity when they access a system.

[1109] "Interaction interface" refers to a display and input system that allows a user and a virtual entity to interact in a virtual space.

[1110] "Prompt sentence" refers to an input sentence provided to an AI model to elicit an appropriate response.

[1111] "Response" refers to the appropriate response that a generative AI model gives in response to user input.

[1112] MODE FOR CARRYING OUT THE INVENTION

[1113] The present invention relates to a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system operates mainly between a server, a terminal, and users.

[1114] First, the user launches the app and selects personal data such as photos, videos, audio files, and text related to the deceased person or pet. The device previews the data and asks for confirmation of upload. Once confirmed, the data is sent from the user's device to the server using the HTTPS protocol. The server stores the received data and notifies the user that it has been successfully received.

[1115] The server then analyzes the stored data, which includes the following steps:

[1116] 1. Image analysis: Use OpenCV to extract facial landmark points and other features from photos and videos.

[1117] 2. Audio analysis: Using the LibROSA library, Mel-Frequency Cepstrum Coefficients (MFCCs) are extracted from the audio data and saved as feature data.

[1118] 3. Text Analysis: Use the NLTK library to tokenize text data and analyze it for frequently occurring words and phrases.

[1119] The server then uses a generative AI model (e.g., GPT-3) based on the extracted feature data to generate a virtual representation of the deceased person or pet, including:

[1120] 1. 3D model generation: Using Blender's API, a 3D model is automatically generated based on facial feature data.

[1121] 2. Voice synthesis: Voice data is generated based on the voice characteristics analyzed by Tacotron 2 and used for virtual characters.

[1122] The generated data is stored in a database for the virtual space, and the server allocates resources for the virtual cemetery and places the generated virtual beings in the virtual space. An access link to the virtual space is generated and sent to the user's terminal. The user's terminal notifies the user that the virtual cemetery is ready and provides the access link.

[1123] When a user clicks on the provided access link, they are given access to the virtual cemetery. Authentication is required for access, and the user enters their authentication information. Once access is authenticated, an interactive interface is displayed.

[1124] When a conversation begins, the server analyzes the message entered by the user and generates an appropriate response. For example, if the user asks, "How have you been?", a response that sounds like the character of the deceased person or pet will be returned (e.g., "I've been fine! How have you been?"). The response from the server is displayed on the user's device, allowing the user to continue the conversation.

[1125] Specific examples

[1126] Data collection

[1127] User: Use the dedicated app to select and upload photos and videos of your pet.

[1128] On your device: Displays "Uploading data" and notifies you "Upload complete" once it's finished.

[1129] Server: Stores the received data and prepares it for analysis.

[1130] Data analysis and generation

[1131] Server: Extracts pet appearance characteristics from photos and videos, and analyzes voice and behavioral patterns from videos and audio.

[1132] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[1133] Preparing for a virtual cemetery

[1134] Server: Allocates resources for the virtual cemetery and generates access links.

[1135] Terminal: Notify the user: "Your virtual graveyard is ready. Please access it at the following link: example_link."

[1136] Dialogue and Interaction

[1137] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[1138] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[1139] Terminal: displays the response from the server and allows the user to continue the interaction.

[1140] As described above, this invention allows users to comprehensively organize data related to deceased loved ones and pets, and based on that data, generate virtual beings and enjoy conversations with them. This system allows users to virtually converse with deceased loved ones and pets, providing peace of mind.

[1141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1142] Step 1: The user launches the dedicated app and selects personal data (photos, videos, audio files, text) related to the deceased person or pet.

[1143] Input: The user operates the dedicated app and specifies personal data.

[1144] Specific operation: The user uses a smartphone or PC to select the relevant file from a folder using the "Data Selection" function within the app.

[1145] Step 2: Your device will preview the selected personal data and ask you to confirm the upload.

[1146] Input: A data file selected by the user.

[1147] Specific behavior: The device will display a preview window to allow the user to check the contents of the data, and a confirmation dialog will appear with the message "Do you want to upload?"

[1148] Step 3: The user confirms the upload and the device sends the data to the server.

[1149] Input: User upload confirmation.

[1150] Output: The data sent to the server.

[1151] What it does: When the user presses the "Upload" button, the device compresses the selected data and sends it to the server using the HTTPS protocol.

[1152] Step 4: The server stores the received data and notifies the user of successful reception.

[1153] Input: Data from the terminal.

[1154] Output: Saved data and notification to the user.

[1155] Specific operation: The server stores the received data in a database and returns an HTTP response saying "Data upload completed." The device receives this response and notifies the user.

[1156] Step 5: The server analyzes the received personal data.

[1157] Input: Stored personal data.

[1158] Output: Extracted feature data.

[1159] Specific behavior:

[1160] Image analysis: Extract facial landmark points from photos and videos using OpenCV.

[1161] Audio analysis: Extract Mel-Frequency Cepstral Coefficients (MFCCs) from audio files using the LibROSA library.

[1162] Text analysis: Use the NLTK library to tokenize text data and analyze frequently occurring words and phrases.

[1163] Step 6: The server uses the extracted feature data to generate a virtual presence of the deceased person or pet using a generative AI model.

[1164] Input: Extracted feature data.

[1165] Output: Data of the generated virtual entity.

[1166] Specific behavior:

[1167] A generative AI model (e.g., GPT-3) is trained on collected text patterns to generate hypothetical conversational responses.

[1168] Using Blender's API, a 3D model is generated based on facial feature data.

[1169] Voice data is generated based on the voice characteristics analyzed by Tacotron 2.

[1170] Step 7: The server stores the generated data in a database for the virtual space and places it in the virtual space.

[1171] Input: Data of the generated virtual entity.

[1172] Output: A virtual entity placed in virtual space.

[1173] How it works: The server uses the Unity server to assign virtual characters to virtual spaces and save the corresponding metadata in a database.

[1174] Step 8: The server generates an access link to the virtual space and sends it to the terminal.

[1175] Input: Virtual space resource information.

[1176] Output: Access link.

[1177] Specific operation: The server returns a dynamically generated URL link to the device in an HTTP response, and the device provides the link to the user.

[1178] Step 9: The user clicks on the provided access link to access the virtual cemetery, enters their authentication information and is authenticated.

[1179] Input: Access link, authentication information.

[1180] Output: Authenticated access to the virtual space.

[1181] Specific operation: The user opens the access link in a browser and enters their user ID and password on the displayed login screen. The server performs authentication, and if successful, they enter the virtual space.

[1182] Step 10: The user inputs a message into the dialogue interface in the virtual space, and the server generates an appropriate response.

[1183] Input: The user's message.

[1184] Output: The appropriate response.

[1185] Specific operation: The server receives the user's message, sends a prompt such as "The user is asking how are you?" to the GPT-3 model, and receives the generated response.

[1186] Step 11: The server sends the generated response to the user terminal and displays it to the user.

[1187] Input: The server-generated response.

[1188] Output: The response displayed on the user's terminal.

[1189] Specific operation: The server sends the generated response as an HTTP response to the user's terminal, and the terminal displays it to the user in text or audio format.

[1190] (Application example 1)

[1191] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1192] Previous virtual presence systems related to deceased loved ones or pets lacked the interactive services that users could experience directly in a physical store. This prevented users from feeling closer to their loved ones or pets through virtual interactions, making it difficult to fully experience peace of mind. Furthermore, without incorporating advanced AI models to generate natural interactions, the user experience was unnatural and did not feel as close to real interactions.

[1193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1194] In this invention, the server includes means for acquiring a plurality of personal data related to the deceased person or pet, means for analyzing the acquired personal data and extracting characteristics of the deceased person or pet, means for generating a virtual presence of the deceased person or pet based on the extracted characteristics, means for providing an interaction interface that allows a user to interact with the virtual presence in a physical store, and means for using a generative AI model to generate natural interactions when the user interacts with the virtual presence, thereby enabling users to interact with the deceased person or pet in a natural way in a physical store and gain peace of mind.

[1195] "Personal data" refers to information such as photos, videos, audio files, and text related to an individual.

[1196] "Features" refer to attributes of the deceased person or pet, such as their appearance, voice, or language usage patterns, extracted from image, audio, and text data.

[1197] "Virtual presence" refers to a virtual character that can be interacted with, including a 3D model and voice, generated from the personal data of a deceased person or pet.

[1198] "Interaction interface" refers to a device, such as a tablet, robot, or smart glasses, that a user uses to communicate with a virtual being.

[1199] A "generative AI model" refers to an artificial intelligence model that generates natural responses to user input.

[1200] "Brick and mortar store" refers to a physical store that a user can actually visit.

[1201] The present invention provides a system that allows users to experience virtual interactions with deceased loved ones or pets in a physical store. This system provides a comprehensive service, from collecting personal data to generating virtual beings and interacting with them through an interface. Specific embodiments are described below.

[1202] Data collection

[1203] Users use dedicated terminals (tablets, robots, or smart glasses) installed in physical stores to input personal data related to the deceased person or pet. This personal data includes photos, videos, audio files, and text. The dedicated terminals then send this data to a server.

[1204] Data analysis and virtual entity generation

[1205] The server analyzes the received personal data. Image data is extracted using DeepFace to identify physical features of the deceased person or pet, while audio data is extracted using Google Text-to-Speech (gTTS) and other voice analysis tools. Text data is analyzed using the OpenAI API to identify frequently used words and speech patterns.

[1206] Based on this feature data, a generative AI model is used to generate a virtual entity, which includes a 3D model and a synthesized voice, and is placed in a virtual space.

[1207] Providing a conversational interface

[1208] The dedicated terminal provides the user with a dialogue interface through which the user can interact with the virtual entity. Questions and comments entered by the user are sent to the server.

[1209] The server analyzes the user's input using a generative AI model and generates an appropriate response. The response is designed to be natural and takes into account the characteristics of the deceased person or pet. The generated response is sent to a dedicated terminal and displayed to the user.

[1210] Specific examples

[1211] For example, if a user uploads photos and videos of their beloved dog "Pochi" in a physical store and asks, "Pochi, how have you been lately?", the server analyzes this data and generates a virtual Pochi. Using a generative AI model, it generates a response to the question, "Pochi, how have you been lately?", returning the response, "I've been fine! How have you been?"

[1212] Prompt Sentence Examples

[1213] Below are some examples of prompt sentences.

[1214] Please reply as Pochi to the following questions: What have you been up to lately?

[1215] The system thus configured allows users to interact with virtual deceased loved ones or pets through natural conversations within a physical store, providing peace of mind.

[1216] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1217] Step 1:

[1218] Users use dedicated terminals (tablets, robots, or smart glasses) installed in physical stores to input personal data such as photos, videos, audio files, and text related to the deceased person or pet. The input data is sent to the server by the terminal. The input is completed when the user selects the data and presses the "upload" button.

[1219] Step 2:

[1220] The server stores the personal data received from the device and then begins analyzing it. Specifically, image data from photos and videos is used to extract physical features of the deceased person or pet using DeepFace. Voice data is used to extract features using Google Text-to-Speech (gTTS) and other voice analysis tools. Text data is analyzed using the OpenAI API to identify frequently used words and speaking patterns.

[1221] Step 3:

[1222] The server uses a generative AI model to generate a virtual presence (3D model and voice) based on the feature data extracted in step 2. This virtual presence reflects the unique characteristics of the deceased person or pet and is capable of natural conversation. The generated virtual presence is stored in a database for the virtual space.

[1223] Step 4:

[1224] The dedicated terminal provides a dialogue interface to the user. The user can use this interface to start a dialogue with the virtual entity. For example, when the user types "Pochi, how have you been lately?", the data is sent from the terminal to the server.

[1225] Step 5:

[1226] The server analyzes the user's questions and comments received in step 4 and generates an appropriate response using the generative AI model. This response is designed to be a natural conversation that takes into account the characteristics of the deceased person or pet. For example, in response to a user's question, "Pochi, how have you been lately?", the server generates a response such as, "I've been fine! How have you been?"

[1227] Step 6:

[1228] The generated response is sent from the server to a dedicated terminal and displayed to the user. The user can confirm the response and continue the dialogue. The content of the dialogue is recorded on the server and used as reference for future dialogues.

[1229] Step 7:

[1230] When the user finishes the conversation, the dedicated terminal closes the conversation interface and the server ends the session. If the user wishes, the conversation content can be saved and a link for the next access can be provided.

[1231] Through the above steps, users can have natural conversations with their deceased loved ones or pets in a physical store, providing peace of mind.

[1232] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1233] The present invention is a system that allows users to create virtual beings based on personal data related to deceased people or pets and engage in conversations in a virtual space. This system incorporates an emotion engine that analyzes the user's emotions, enabling more personalized conversations. The system operates primarily between a server, a terminal, and the user.

[1234] User Data Collection

[1235] 1. Data Collection

[1236] User: The user launches the app and selects personal data related to the deceased person or pet, including photos, videos, audio files, and text.

[1237] On the device: The device will preview the data you selected and ask you to confirm the upload. Once confirmed, the data will be uploaded to the server.

[1238] Server: The server stores the received data and notifies the user that it was received successfully.

[1239] Data analysis and generation

[1240] 2. Data Analysis

[1241] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[1242] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[1243] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[1244] Server: Stores these feature data in a database.

[1245] 3. Virtual Being Creation

[1246] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[1247] Server: Stores the generated virtual entities in a database for the virtual space.

[1248] Preparing for a virtual cemetery

[1249] 4. Resource Allocation and Link Creation

[1250] Server: Allocates resources for the virtual graveyard and places the generated virtual beings in the virtual space.

[1251] Server: Generates a link to access the virtual cemetery and sends it to the device.

[1252] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[1253] Use of dialogue and emotion engines

[1254] 5. Access and Authentication

[1255] User: Clicks on the provided access link to access the virtual cemetery.

[1256] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[1257] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[1258] 6. Start a dialogue

[1259] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[1260] Terminal: Sends user input to the server.

[1261] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[1262] Server: Sends the generated response to the terminal.

[1263] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[1264] Use of emotion engine

[1265] 7. Emotional Recognition

[1266] Device: The device is equipped with facial recognition and voice analysis functions, and analyzes the user's emotions in real time from their facial expressions and voice.

[1267] Server: Receives the analysis results and sends them to the emotion engine.

[1268] Server: The emotion engine adjusts the content and tone of the response based on the user's emotional data.

[1269] 8. Personalized responses

[1270] Server: Generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it will return words of encouragement.

[1271] Server: Sends the generated response to the terminal.

[1272] Terminal: Displays the response received from the server to the user, for example, "Cheer up, we're always thinking about you."

[1273] Specific examples

[1274] As an example, we will explain how a user can upload photos or videos of a deceased pet and start a dialogue using the emotion engine.

[1275] 1. Data Collection

[1276] User: Use the dedicated app to select and upload photos and videos of your pet.

[1277] On your device: Display the progress of the data upload and notify you when it's finished by saying "Upload complete."

[1278] Server: Stores the received data and prepares it for analysis.

[1279] 2. Data Analysis and Generation

[1280] Server: Extracts the pet's appearance characteristics from photos and videos and analyzes its voice characteristics.

[1281] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[1282] 3. Preparing a Virtual Graveyard

[1283] Server: Allocates resources for the virtual cemetery and generates access links.

[1284] Terminal: Send the link to the user and tell them, "Your virtual cemetery is ready. Please access it from the following link: example_link."

[1285] 4. Use of dialogue and emotion engines

[1286] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[1287] Terminal: Sends user input to the server.

[1288] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[1289] Terminal: Displays the response from the server and shows it to the user.

[1290] Terminal: Analyzes emotions from the user's facial expressions and voice, and sends the results to the server.

[1291] Server: The emotion engine generates a response based on the user's emotions, for example, returning encouraging words such as "You're sad, we're thinking of you too."

[1292] The system allows users to virtually interact with deceased loved ones or pets and receive personalized responses based on their emotions, providing greater peace of mind.

[1293] The processing flow will be explained below.

[1294] Program processing steps

[1295] Step 1: Select and upload data

[1296] User: Launch the dedicated app and select photos, videos, audio, and text data of the deceased person or pet.

[1297] On the device: Preview the selected file and ask the user for confirmation. If the user presses the confirm button, the file will be uploaded to the server.

[1298] Server: Stores the received data in a database and notifies the device that the upload is complete.

[1299] Terminal: Notify the user that the upload was successful.

[1300] Step 2: Data analysis

[1301] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[1302] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[1303] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[1304] Server: Stores these feature data in a database.

[1305] Step 3: Creating a Virtual Being

[1306] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[1307] Server: Stores the generated virtual entities in a database for the virtual space.

[1308] Step 4: Preparing the Virtual Graveyard and Creating Links

[1309] Server: Allocates resources for the virtual graveyard and places the generated virtual beings in the virtual space.

[1310] Server: Generates a link to access the virtual cemetery and sends it to the device.

[1311] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[1312] Step 5: Access and Authentication

[1313] User: Clicks on the provided access link to access the virtual cemetery.

[1314] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[1315] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[1316] Step 6: Start a conversation

[1317] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[1318] Terminal: Sends user input to the server.

[1319] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[1320] Server: Sends the generated response to the terminal.

[1321] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[1322] Step 7: Recognize emotions

[1323] Device: The device is equipped with facial recognition and voice analysis functions, and analyzes the user's emotions in real time from their facial expressions and voice.

[1324] Server: Receives the analysis results and sends them to the emotion engine.

[1325] Server: The emotion engine adjusts the content and tone of the response based on the user's emotional data.

[1326] Step 8: Personalize your response

[1327] Server: Generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it will return words of encouragement.

[1328] Server: Sends the generated response to the terminal.

[1329] Terminal: Displays the response received from the server to the user, for example, "Cheer up, we're always thinking about you."

[1330] Step 9: Additional Interactions

[1331] User: Continues to reminisce about and ask questions about deceased loved ones and pets. Performs specific actions in the virtual space, such as placing flowers.

[1332] Terminal: Sends user actions to the server and displays the results of the execution.

[1333] Server: Updates the virtual space in real time based on the user's actions, allowing the user to continue interacting.

[1334] In this way, users can virtually converse and interact with their deceased loved ones or pets, bringing them peace of mind.

[1335] Example 2

[1336] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1337] Today, many people use photos and videos to remember their deceased loved ones and pets, but these alone are not enough to bring them peace of mind. While there are systems that allow users to virtually interact with their deceased loved ones and pets, few systems can respond to emotional changes in real time. As a result, it is currently difficult for users to receive personalized experiences tailored to their emotions.

[1338] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to select multiple pieces of personal data related to the deceased person or pet; means for transmitting the selected personal data to the server; means for analyzing the personal data received by the server and extracting characteristics of the deceased person or pet; means for generating a virtual presence of the deceased person or pet using a generative AI model based on the extracted characteristics; means for placing the generated virtual presence in a virtual space; means for generating and providing a link for accessing the virtual space to the user; means for the user to interact with the virtual presence in the virtual space; an emotion engine for analyzing the user's emotions in real time; and means for personalizing responses based on the emotion engine. This allows the user to receive personalized responses according to their emotions while virtually interacting with the deceased person or pet.

[1339] "User" refers to an individual who uses this system.

[1340] "Personal data" refers to information such as photos, videos, audio files, and text related to the deceased person or pet.

[1341] "Terminal" refers to a device operated by a user, such as a computer, smartphone, or tablet.

[1342] "Server" refers to the computer that processes and stores data at the center of this system.

[1343] "Analysis" refers to the process of extracting characteristics and information about the deceased person or pet from personal data.

[1344] A "generative AI model" refers to an artificial intelligence model that uses machine learning algorithms to generate new data or formats.

[1345] "Virtual presence" refers to a digital representation of a deceased person or pet created by a generative AI model based on analyzed personal data.

[1346] "Virtual space" refers to a digital environment in which users interact with virtual beings.

[1347] "Link" refers to a URL or hyperlink for accessing a virtual space.

[1348] An "emotion engine" refers to a system that analyzes a user's emotions and generates a response based on those emotions.

[1349] The present invention provides a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system is mainly composed of a server, terminals, and users.

[1350] User Data Collection

[1351] User: Launches the dedicated app and selects photos, videos, audio files, and text files related to the deceased person or pet.

[1352] On the device: Preview the selected data and ask the user to confirm the upload. Once confirmed, send the data to the server.

[1353] Server: Saves the received data in local storage or cloud storage and notifies the user that the upload is complete.

[1354] Data analysis and generation

[1355] Server: Analyzes the received photos and videos using image analysis algorithms (e.g., OpenCV or TensorFlow) to extract the external characteristics of the deceased person or pet.

[1356] Server: The audio file is analyzed using a speech analysis algorithm (e.g., Google Cloud Speech-to-Text or IBM Watson) to extract voice characteristics.

[1357] Server: Analyzes the text data using natural language processing algorithms (e.g., spaCy or BERT) to extract frequently used words and distinctive language patterns.

[1358] Server: Stores these feature data in a database (e.g., PostgreSQL or MongoDB).

[1359] Server: Based on the stored feature data, a generative AI model (e.g., GPT-3 or StyleGAN) is used to generate virtual deceased people or pets. This generation process includes creating 3D models and synthesizing voices. The generated virtual beings are stored in a database for the virtual space.

[1360] Preparing for a virtual cemetery

[1361] Server: Allocate resources for the virtual cemetery and place the virtual presence of deceased people and pets in the virtual space. This includes securing cloud server resources and generating the virtual space.

[1362] Server: Generates and provides users with a dedicated link to access the virtual cemetery.

[1363] Terminal: Notifies the user of the generated access link and informs them that the virtual cemetery is ready.

[1364] Use of dialogue and emotion engines

[1365] User: Clicks on the provided access link to access the virtual cemetery.

[1366] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[1367] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[1368] User: Initiate a virtual conversation with a deceased loved one or pet, for example, by typing "How have you been?"

[1369] Terminal: Sends user input to the server.

[1370] Server: Analyzes the user's input using an AI model, generates responses that the deceased person or pet would likely make, and sends them to the device.

[1371] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[1372] Use of emotion engine

[1373] Device: Using facial recognition and voice analysis functions, emotions are analyzed in real time from the user's facial expressions and voice.

[1374] Server: Receives the analysis results and sends them to the emotion engine.

[1375] Server: The emotion engine generates personalized responses by adjusting the content and tone of the response based on the user's emotional data. For example, if the user is sad, it adds words of encouragement.

[1376] Server: Sends the generated response to the terminal and displays it to the user, for example, "Cheer up, I'm always worried about you."

[1377] Specific examples

[1378] For example, consider a user uploading photos and videos of a deceased pet to use the system:

[1379] 1. User: Use the dedicated app to select and upload photos and videos of your pet.

[1380] 2. Device: Displays the progress of data upload and notifies you when it is complete.

[1381] 3. Server: Stores the received data and prepares it for analysis.

[1382] 4. Server: Extracts the pet's external features from photos and videos and analyzes its audio features.

[1383] 5. Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[1384] 6. Server: Allocates resources for the virtual cemetery and generates access links.

[1385] 7. Terminal: Notify the user of the link and say, "Your virtual cemetery is ready. Please access it from the following link: example_link."

[1386] 8. User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[1387] 9. Terminal: Sends user input to the server and displays responses to the user. It displays responses such as "Pochi was fine! How was yours?"

[1388] 10. Terminal: Analyzes emotions from the user's facial expressions and voice and sends them to the server.

[1389] 11. Server: The emotion engine generates a response based on the user's emotions, returning encouraging words such as "You're sad, we're thinking of you too." This system allows the user to find greater peace of mind.

[1390] This Detailed Description provides a detailed understanding of how the present invention may be practiced.

[1391] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1392] Program processing steps

[1393] Step 1: Collect user data

[1394] Input: The user selects photos, videos, audio files, and text related to the deceased person or pet.

[1395] How it works: Launch the dedicated app and select data through the interface.

[1396] Output: The selected data is saved to the device and sent to the next processing step.

[1397] Step 2: Upload data

[1398] Input: Personal data stored on your device.

[1399] How it works: The device previews the selected data and asks the user to confirm the upload. Once confirmed, the data is sent to the server.

[1400] Output: Personal data is uploaded to the server.

[1401] Step 3: Data storage and notification

[1402] Input: Personal data received by the server.

[1403] How it works: The server saves the received data to local storage or cloud storage and notifies the user that the upload is complete.

[1404] Output: The data is saved and the user is notified that the upload is complete.

[1405] Step 4: Image analysis

[1406] Input: Photos and videos from your saved personal data.

[1407] How it works: The server uses image analysis algorithms (e.g., OpenCV or TensorFlow) to extract the external features of the deceased person or pet.

[1408] Output: Appearance feature data is generated.

[1409] Step 5: Audio analysis

[1410] Input: Audio file from saved personal data.

[1411] How it works: The server uses voice analysis algorithms (e.g., Google Cloud Speech-to-Text or IBM Watson) to extract voice characteristics and tone.

[1412] Output: Voice feature data is generated.

[1413] Step 6: Text Analysis

[1414] Input: A text file from your saved personal data.

[1415] How it works: The server uses natural language processing algorithms (e.g. spaCy or BERT) to analyze frequently used words and unique language patterns.

[1416] Output: Word and language pattern data is generated.

[1417] Step 7: Save to database

[1418] Input: Analyzed appearance feature data, voice feature data, and word and language pattern data.

[1419] How it works: The server stores this data in a database (e.g. PostgreSQL or MongoDB).

[1420] Output: Feature data stored in a database.

[1421] Step 8: Virtual Being Generation

[1422] Input: Stored feature data.

[1423] How it works: The server uses generative AI models (e.g., GPT-3 or StyleGAN) to generate virtual versions of deceased people and pets, including creating 3D models and synthesizing voices.

[1424] Output: Virtual entity data is generated.

[1425] Step 9: Virtual Space Placement

[1426] Input: Data of the generated virtual being.

[1427] How it works: The server allocates resources for the virtual cemetery and places virtual figures of deceased people and pets in the virtual space.

[1428] Output: A virtual presence of a deceased person or pet placed in a virtual space.

[1429] Step 10: Link generation and notification

[1430] Input: A virtual entity placed in a virtual space.

[1431] How it works: The server generates a dedicated link to access the virtual cemetery and provides it to the user.

[1432] Output: The generated access link is notified to the user.

[1433] Step 11: Authentication and Access

[1434] Input: The access link the user received.

[1435] How it works: The user clicks on the provided link to access the virtual cemetery. The device sends authentication information to the server, which then performs the authentication process.

[1436] Output: The user can now access the virtual graveyard.

[1437] Step 12: Start a conversation

[1438] Input: A prompt that the user types in the virtual space (e.g., "How have you been?").

[1439] How it works: The device sends user input to the server, which then analyzes the prompt and uses a generative AI model to generate a response for the deceased person or pet. The generated response is then sent to the device.

[1440] Output: The user will see a response from the deceased person or pet (e.g., "I'm fine! How have you been?").

[1441] Step 13: Emotion Recognition

[1442] Input: The user's facial expressions and voice during the conversation.

[1443] How it works: The device uses facial recognition and voice analysis to analyze the user's facial expressions and voice in real time.

[1444] Output: The parsed emotion data is generated.

[1445] Step 14: Personalize your response

[1446] Input: Parsed emotion data and the following user input:

[1447] Operation: The server uses the emotion engine to generate a response based on the user's emotion. The generated response is sent to the device.

[1448] Output: A personalized response is displayed to the user (e.g., "Cheer up, we're always worried about you").

[1449] This allows the user to enjoy a virtual conversation while receiving individual responses according to their emotions.

[1450] (Application example 2)

[1451] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1452] Conventional virtual dialogue systems have the problem that interactions with virtual beings generated based on personal data related to deceased loved ones or pets cannot flexibly respond to the user's emotions or circumstances, which limits the peace of mind that users can gain and makes it difficult to achieve personalized dialogue.

[1453] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for importing multiple pieces of personal data related to the deceased person or pet, means for analyzing the imported personal data and extracting characteristics of the deceased person or pet, means for generating a virtual presence of the deceased person or pet based on the extracted characteristics, means for providing a virtual space in which the user can interact with the generated virtual presence, means including an emotion engine for analyzing the user's emotions, and means for adjusting the content and tone of the response based on the analyzed emotion data. This enables personalized dialogue according to the user's emotions and situation.

[1454] "Deceased person or pet" refers to an object that the user recognizes as a deceased person or pet and recreates in the virtual space.

[1455] "Personal Data" refers to data containing personal information, such as photographs, videos, audio files, and text related to a deceased person or pet.

[1456] "Means for extracting features" refers to the processes and techniques for analyzing and extracting the appearance, vocal characteristics, and language patterns of the deceased person or pet from the collected personal data.

[1457] "Means for generating virtual entities" refers to the technology and process for generating virtual entities, such as 3D models and voices of deceased individuals or pets, based on extracted feature data.

[1458] "Virtual space" refers to a digitally simulated environment in which users can interact with generated virtual beings.

[1459] An "emotion engine" refers to software and algorithms that analyze a user's emotions from their facial expressions and voice and adjust the content and tone of their responses accordingly.

[1460] "Means for providing an interactive virtual space" refers to the technologies and processes for constructing and making accessible a virtual space that allows users to interact with generated virtual beings.

[1461] "Emotion data" refers to information about emotions obtained by analyzing the user's facial expressions and voice.

[1462] "Response content" refers to the content of a reply to an input made by a user to a virtual entity.

[1463] "Tone of response" refers to the nuances and tone of voice of the response to the user that are adjusted by the emotion engine.

[1464] An embodiment of the present invention will be described in detail below. The present invention is a system that allows a user to generate a virtual entity based on personal data related to a deceased person or pet and to engage in a dialogue in a virtual space. The system incorporates an emotion engine and includes a function for analyzing the user's emotions and providing a more personalized dialogue.

[1465] Users use a dedicated application to collect personal data related to the deceased person or pet. This application is installed on a smartphone and allows users to upload data such as photos, videos, audio files, and text. The device previews the data selected by the user, asks for confirmation of the upload, and once confirmation is received, sends the data to the server. The server stores the received data and notifies the user that it has been successfully received.

[1466] Using the generative AI model, the server analyzes the received data and extracts the characteristics of the deceased person or pet. Specifically, it uses an image analysis algorithm to extract external features from photos and videos, and an audio analysis algorithm to extract voice characteristics and tone from audio data. It also analyzes frequently used words and unique language patterns from text data and stores this feature data in a database. Based on the analyzed feature data, the server creates a 3D model and synthesizes voice to generate a virtual deceased person or pet.

[1467] The generated virtual beings are placed in a virtual space. The server allocates resources for the virtual space, places the generated virtual beings in the virtual space, and generates a link to access the virtual cemetery. This link is provided to the user via the terminal, and the user clicks on the provided link to access the virtual cemetery. The terminal sends the user's authentication information to the server, and if the authentication is successful, a dialogue system is launched and guides the user to the virtual cemetery.

[1468] Users can interact with their deceased loved ones or pets in a virtual space. For example, by inputting a question such as "How have you been?", the device sends the input to the server, which then analyzes it using a generative AI model to generate a response that the deceased or pet would likely give. The generated response is then sent to the device and displayed to the user.

[1469] The device also provides a function that analyzes the user's emotions in real time using an emotion engine. The device is equipped with facial recognition and voice analysis functions, which analyze the user's emotions from their facial expressions and voice and send the results to the server. The server then uses the emotion engine to adjust the content and tone of the response based on the analysis results, generating a personalized response that matches the user's emotions.

[1470] Here are some examples of prompts:

[1471] User: Pochi, what have you been up to lately?

[1472] System: Pochi was fine! How were you?

[1473] The system allows users to virtually interact with deceased loved ones or pets and receive personalized responses based on their emotions, providing greater peace of mind.

[1474] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1475] Step 1:

[1476] The user launches a dedicated application and selects personal data (photos, videos, audio files, text, etc.) related to the deceased person or pet. The application prompts the user to upload the selected data. The application previews the data selected by the user and asks for confirmation of the upload. Once the user confirms, the device sends the data to the server. The input is the personal data selected by the user, and the output is the transmission of the data to the server.

[1477] Step 2:

[1478] The server stores the received personal data and notifies the user that it has been received successfully. The server then uses image analysis algorithms to analyze the photos and videos to extract the physical characteristics of the deceased person or pet. It also uses voice analysis algorithms to analyze the audio data to extract voice characteristics and tone. It also extracts frequently used words and distinctive language patterns from the text data. The input is the personal data uploaded to the server, and the output is data on physical characteristics, voice characteristics, and language patterns.

[1479] Step 3:

[1480] The server uses a generative AI model to generate a virtual deceased person or pet based on the analyzed feature data. This generation process includes creating a 3D model and synthesizing voice. The data of the generated virtual being is stored in a database for the virtual space. The input is the extracted feature data, and the output is the data of the generated virtual being.

[1481] Step 4:

[1482] The server allocates resources for the virtual cemetery and places the generated virtual beings in the virtual space. Next, it generates a link to access the virtual cemetery and sends it to the terminal. The terminal notifies the user of this link and informs them that the virtual cemetery is ready. The input is the data of the virtual beings, and the output is the generation and notification of the access link.

[1483] Step 5:

[1484] The user clicks on the provided access link to access the virtual cemetery. The terminal sends the user's authentication information to the server and executes the authentication process. If the authentication is successful, the server launches a dialogue system and guides the user to the virtual cemetery. The input is the user's access link and authentication information, and the output is access to the virtual cemetery and the launch of the dialogue system.

[1485] Step 6:

[1486] The user interacts with the deceased person or pet in the virtual space. For example, they might type, "How have you been?" The device sends the user's input to the server. The server analyzes the user's input using a generative AI model and generates a response that the deceased person or pet would likely give. The generated response is sent to the device and displayed to the user. The input is the user's interaction, and the output is the generated response.

[1487] Step 7:

[1488] The device uses facial recognition and voice analysis functions to analyze emotions from the user's facial expressions and voice. The analysis results are sent to the server. The server's emotion engine adjusts the response content and tone based on the user's emotional data. The input is the user's facial expressions and voice data, and the output is the adjusted response content and tone.

[1489] Step 8:

[1490] The server generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it generates words of encouragement and sends the generated response to the device, which then displays it to the user. The input is the analysis results of the emotion engine, and the output is a personalized response.

[1491] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1492] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1493] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1494] [Fourth embodiment]

[1495] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1496] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1497] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1498] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1499] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1500] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1501] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1502] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1503] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1504] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1505] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1506] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1507] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1508] The present invention provides a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system operates mainly between a server, a terminal, and users.

[1509] User Data Collection

[1510] 1. Data Collection

[1511] User: The user launches the app and selects personal data related to the deceased person or pet, including photos, videos, audio files, and text.

[1512] On your device: The device will preview the data you selected and ask you to confirm the upload. Once confirmed, the data will be sent to the server.

[1513] Server: The server stores the received data and notifies the user that it was received successfully.

[1514] Data analysis and generation

[1515] 2. Data Analysis

[1516] Server: The server analyzes the received personal data. It uses image analysis algorithms to extract features of the deceased person or pet from photos and videos, analyzes audio data to extract voice characteristics, and analyzes text data to identify frequently used words and speaking patterns.

[1517] 3. Virtual Being Creation

[1518] Server: Based on the extracted feature data, a generative AI model is used to generate a virtual presence of the deceased person or pet, including 3D model generation and voice synthesis.

[1519] Server: Stores the generated data in a database for the virtual space.

[1520] Preparing for a virtual cemetery

[1521] 4. Resource Allocation and Link Creation

[1522] Server: Allocates resources for the virtual graveyard, places the generated virtual beings in the virtual space, generates access links to the virtual space, and sends them to the terminals.

[1523] Terminal: The terminal notifies the user that the virtual cemetery is ready and provides an access link.

[1524] Dialogue and Interaction

[1525] 5. Access and Interaction

[1526] User: The user clicks on the provided access link to access the virtual cemetery. Authentication is required for access, ensuring security.

[1527] Terminal: Once access to the virtual graveyard is authenticated, an interactive interface is displayed.

[1528] 6. Start a dialogue

[1529] Server: Analyzes the message entered by the user and generates an appropriate response. For example, the question "How have you been?" will be answered by a character that resembles a deceased person or pet.

[1530] Terminal: Receives the response from the server and displays it to the user.

[1531] Specific examples

[1532] As an example, we will explain the process when a user uploads photos and videos of a deceased pet.

[1533] 1. Data Collection

[1534] User: Use the dedicated app to select and upload photos and videos of your pet.

[1535] On your device: Displays "Uploading data" and notifies you "Upload complete" once it's finished.

[1536] Server: Stores the received data and prepares it for analysis.

[1537] 2. Data Analysis and Generation

[1538] Server: Extracts pet appearance characteristics from photos and videos, and analyzes voice and behavioral patterns from videos and audio.

[1539] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[1540] 3. Preparing a Virtual Graveyard

[1541] Server: Allocates resources for the virtual cemetery and generates access links.

[1542] Terminal: Notify the user: "Your virtual graveyard is ready. Please access it at the following link: example_link."

[1543] 4. Dialogue and Interaction

[1544] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[1545] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[1546] Terminal: displays the response from the server and allows the user to continue the interaction.

[1547] The system allows users to virtually interact with deceased loved ones or pets, providing peace of mind.

[1548] The processing flow will be explained below.

[1549] Program processing steps

[1550] Step 1: Select and upload data

[1551] User: Launch the dedicated app and select photos, videos, audio, and text data of the deceased person or pet.

[1552] On the device: Preview the selected file and ask the user for confirmation. If the user presses the confirm button, the file will be uploaded to the server.

[1553] Server: Stores the received data in a database and notifies the device that the upload is complete.

[1554] Terminal: Notify the user that the upload was successful.

[1555] Step 2: Data analysis

[1556] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[1557] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[1558] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[1559] Server: Stores these feature data in a database.

[1560] Step 3: Creating a Virtual Being

[1561] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[1562] Server: Stores the generated virtual entities in a database for the virtual space.

[1563] Step 4: Preparing the Virtual Graveyard and Creating Links

[1564] Server: Allocates virtual graveyard resources and places the generated virtual beings in the virtual space.

[1565] Server: Generates a link to access the virtual cemetery and sends it to the device.

[1566] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[1567] Step 5: Access and Authentication

[1568] User: Clicks on the provided access link to access the virtual cemetery.

[1569] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[1570] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[1571] Step 6: Start a conversation

[1572] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[1573] Terminal: Sends user input to the server.

[1574] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[1575] Server: Sends the generated response to the terminal.

[1576] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[1577] Step 7: Additional Interactions

[1578] User: Continues to reminisce about and ask questions about deceased loved ones and pets. Performs specific actions in the virtual space, such as placing flowers.

[1579] Terminal: Sends user actions to the server and displays the results of the execution.

[1580] Server: Updates the virtual space in real time based on the user's actions, allowing the user to continue interacting.

[1581] In this way, users can virtually converse and interact with their deceased loved ones or pets, bringing them peace of mind.

[1582] Example 1

[1583] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1584] In modern society, people cherish memories of their deceased loved ones and pets and wish to feel close to them forever. However, existing technologies lack the means to recreate the presence of deceased people or pets in a virtual space and provide them in an interactive form using personal data related to them. Therefore, there is a need for a system that analyzes data related to deceased people or pets, generates virtual presences of the deceased or pets based on that data, and allows users to interact with them.

[1585] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1586] In this invention, the server includes: means for a user to select multiple pieces of personal data related to the deceased person or pet; means for previewing the selected personal data on a user terminal and confirming upload; means for transmitting the confirmed personal data to the server and saving it; means for analyzing the saved personal data and extracting characteristics of the deceased person or pet; means for generating a virtual being of the deceased person or pet using a generative AI model based on the extracted characteristic data; means for generating a 3D model of the generated virtual being and performing voice synthesis; means for storing the generated virtual being in a virtual space database and placing it in the virtual space; means for generating an access link to the virtual space and sending it to the user terminal; means for a user to access the virtual space and be authenticated by entering authentication information; means for enabling interaction with the virtual being in the virtual space if authentication is successful; means for analyzing messages from the interacting user and generating appropriate responses; and means for displaying the generated responses on the user terminal. This allows users to comprehensively organize data related to the deceased person or pet, generate virtual beings based on the data, and enjoy interacting with them.

[1587] "User" refers to a person who uses the system to manipulate personal data related to a deceased person or pet and to interact in a virtual space.

[1588] "Personal data" refers to digital information such as photos, videos, audio files, and text related to a deceased person or pet.

[1589] "User terminal" refers to a device used by a user to select, preview, review, and transmit personal data.

[1590] "Server" refers to a central computer system that receives, stores, and analyzes personal data sent by users and generates a virtual presence.

[1591] An "image analysis algorithm" refers to a computational method for extracting specific features from photos and videos.

[1592] "Speech analysis" refers to the process of extracting speech features from speech data.

[1593] "Text analytics" refers to the process of analyzing text data for frequently used words and speech patterns.

[1594] "Generative AI model" refers to an AI (artificial intelligence) model that uses machine learning technology to generate virtual versions of deceased people or pets.

[1595] "3D model generation" refers to the process of creating a three-dimensional virtual model based on characteristic data of a deceased person or pet.

[1596] "Speech synthesis" refers to the process of generating new speech from analyzed speech data.

[1597] "Database for virtual space" refers to a database for storing data on generated virtual entities.

[1598] "Access link" refers to a URL or the like that allows a user to access a virtual space.

[1599] "Authentication" refers to the process of verifying a user's identity when they access a system.

[1600] "Interaction interface" refers to a display and input system that allows a user and a virtual entity to interact in a virtual space.

[1601] "Prompt sentence" refers to an input sentence provided to an AI model to elicit an appropriate response.

[1602] "Response" refers to the appropriate response that a generative AI model gives in response to user input.

[1603] MODE FOR CARRYING OUT THE INVENTION

[1604] The present invention relates to a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system operates mainly between a server, a terminal, and users.

[1605] First, the user launches the app and selects personal data such as photos, videos, audio files, and text related to the deceased person or pet. The device previews the data and asks for confirmation of upload. Once confirmed, the data is sent from the user's device to the server using the HTTPS protocol. The server stores the received data and notifies the user that it has been successfully received.

[1606] The server then analyzes the stored data, which includes the following steps:

[1607] 1. Image analysis: Use OpenCV to extract facial landmark points and other features from photos and videos.

[1608] 2. Audio analysis: Using the LibROSA library, Mel-Frequency Cepstrum Coefficients (MFCCs) are extracted from the audio data and saved as feature data.

[1609] 3. Text Analysis: Use the NLTK library to tokenize text data and analyze it for frequently occurring words and phrases.

[1610] The server then uses a generative AI model (e.g., GPT-3) based on the extracted feature data to generate a virtual representation of the deceased person or pet, including:

[1611] 1. 3D model generation: Using Blender's API, a 3D model is automatically generated based on facial feature data.

[1612] 2. Voice synthesis: Voice data is generated based on the voice characteristics analyzed by Tacotron 2 and used for virtual characters.

[1613] The generated data is stored in a database for the virtual space, and the server allocates resources for the virtual cemetery and places the generated virtual beings in the virtual space. An access link to the virtual space is generated and sent to the user's terminal. The user's terminal notifies the user that the virtual cemetery is ready and provides the access link.

[1614] When a user clicks on the provided access link, they are given access to the virtual cemetery. Authentication is required for access, and the user enters their authentication information. Once access is authenticated, an interactive interface is displayed.

[1615] When a conversation begins, the server analyzes the message entered by the user and generates an appropriate response. For example, if the user asks, "How have you been?", a response that sounds like the character of the deceased person or pet will be returned (e.g., "I've been fine! How have you been?"). The response from the server is displayed on the user's device, allowing the user to continue the conversation.

[1616] Specific examples

[1617] Data collection

[1618] User: Use the dedicated app to select and upload photos and videos of your pet.

[1619] On your device: Displays "Uploading data" and notifies you "Upload complete" once it's finished.

[1620] Server: Stores the received data and prepares it for analysis.

[1621] Data analysis and generation

[1622] Server: Extracts pet appearance characteristics from photos and videos, and analyzes voice and behavioral patterns from videos and audio.

[1623] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[1624] Preparing for a virtual cemetery

[1625] Server: Allocates resources for the virtual cemetery and generates access links.

[1626] Terminal: Notify the user: "Your virtual graveyard is ready. Please access it at the following link: example_link."

[1627] Dialogue and Interaction

[1628] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[1629] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[1630] Terminal: displays the response from the server and allows the user to continue the interaction.

[1631] As described above, this invention allows users to comprehensively organize data related to deceased loved ones and pets, and based on that data, generate virtual beings and enjoy conversations with them. This system allows users to virtually converse with deceased loved ones and pets, providing peace of mind.

[1632] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1633] Step 1: The user launches the dedicated app and selects personal data (photos, videos, audio files, text) related to the deceased person or pet.

[1634] Input: The user operates the dedicated app and specifies personal data.

[1635] Specific operation: The user uses a smartphone or PC to select the relevant file from a folder using the "Data Selection" function within the app.

[1636] Step 2: Your device will preview the selected personal data and ask you to confirm the upload.

[1637] Input: A data file selected by the user.

[1638] Specific behavior: The device will display a preview window to allow the user to check the contents of the data, and a confirmation dialog will appear with the message "Do you want to upload?"

[1639] Step 3: The user confirms the upload and the device sends the data to the server.

[1640] Input: User upload confirmation.

[1641] Output: The data sent to the server.

[1642] What it does: When the user presses the "Upload" button, the device compresses the selected data and sends it to the server using the HTTPS protocol.

[1643] Step 4: The server stores the received data and notifies the user of successful reception.

[1644] Input: Data from the terminal.

[1645] Output: Saved data and notification to the user.

[1646] Specific operation: The server stores the received data in a database and returns an HTTP response saying "Data upload completed." The device receives this response and notifies the user.

[1647] Step 5: The server analyzes the received personal data.

[1648] Input: Stored personal data.

[1649] Output: Extracted feature data.

[1650] Specific behavior:

[1651] Image analysis: Extract facial landmark points from photos and videos using OpenCV.

[1652] Audio analysis: Extract Mel-Frequency Cepstral Coefficients (MFCCs) from audio files using the LibROSA library.

[1653] Text analysis: Use the NLTK library to tokenize text data and analyze frequently occurring words and phrases.

[1654] Step 6: The server uses the extracted feature data to generate a virtual presence of the deceased person or pet using a generative AI model.

[1655] Input: Extracted feature data.

[1656] Output: Data of the generated virtual entity.

[1657] Specific behavior:

[1658] A generative AI model (e.g., GPT-3) is trained on collected text patterns to generate hypothetical conversational responses.

[1659] Using Blender's API, a 3D model is generated based on facial feature data.

[1660] Voice data is generated based on the voice characteristics analyzed by Tacotron 2.

[1661] Step 7: The server stores the generated data in a database for the virtual space and places it in the virtual space.

[1662] Input: Data of the generated virtual entity.

[1663] Output: A virtual entity placed in virtual space.

[1664] How it works: The server uses the Unity server to assign virtual characters to virtual spaces and save the corresponding metadata in a database.

[1665] Step 8: The server generates an access link to the virtual space and sends it to the terminal.

[1666] Input: Virtual space resource information.

[1667] Output: Access link.

[1668] Specific operation: The server returns a dynamically generated URL link to the device in an HTTP response, and the device provides the link to the user.

[1669] Step 9: The user clicks on the provided access link to access the virtual cemetery, enters their authentication information and is authenticated.

[1670] Input: Access link, authentication information.

[1671] Output: Authenticated access to the virtual space.

[1672] Specific operation: The user opens the access link in a browser and enters their user ID and password on the displayed login screen. The server performs authentication, and if successful, they enter the virtual space.

[1673] Step 10: The user inputs a message into the dialogue interface in the virtual space, and the server generates an appropriate response.

[1674] Input: The user's message.

[1675] Output: The appropriate response.

[1676] Specific operation: The server receives the user's message, sends a prompt such as "The user is asking how are you?" to the GPT-3 model, and receives the generated response.

[1677] Step 11: The server sends the generated response to the user terminal and displays it to the user.

[1678] Input: The server-generated response.

[1679] Output: The response displayed on the user's terminal.

[1680] Specific operation: The server sends the generated response as an HTTP response to the user's terminal, and the terminal displays it to the user in text or audio format.

[1681] (Application example 1)

[1682] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1683] Previous virtual presence systems related to deceased loved ones or pets lacked the interactive services that users could experience directly in a physical store. This prevented users from feeling closer to their loved ones or pets through virtual interactions, making it difficult to fully experience peace of mind. Furthermore, without incorporating advanced AI models to generate natural interactions, the user experience was unnatural and did not feel as close to real interactions.

[1684] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1685] In this invention, the server includes means for acquiring a plurality of personal data related to the deceased person or pet, means for analyzing the acquired personal data and extracting characteristics of the deceased person or pet, means for generating a virtual presence of the deceased person or pet based on the extracted characteristics, means for providing an interaction interface that allows a user to interact with the virtual presence in a physical store, and means for using a generative AI model to generate natural interactions when the user interacts with the virtual presence, thereby enabling users to interact with the deceased person or pet in a natural way in a physical store and gain peace of mind.

[1686] "Personal data" refers to information such as photos, videos, audio files, and text related to an individual.

[1687] "Features" refer to attributes of the deceased person or pet, such as their appearance, voice, or language usage patterns, extracted from image, audio, and text data.

[1688] "Virtual presence" refers to a virtual character that can be interacted with, including a 3D model and voice, generated from the personal data of a deceased person or pet.

[1689] "Interaction interface" refers to a device, such as a tablet, robot, or smart glasses, that a user uses to communicate with a virtual being.

[1690] A "generative AI model" refers to an artificial intelligence model that generates natural responses to user input.

[1691] "Brick and mortar store" refers to a physical store that a user can actually visit.

[1692] The present invention provides a system that allows users to experience virtual interactions with deceased loved ones or pets in a physical store. This system provides a comprehensive service, from collecting personal data to generating virtual beings and interacting with them through an interface. Specific embodiments are described below.

[1693] Data collection

[1694] Users use dedicated terminals (tablets, robots, or smart glasses) installed in physical stores to input personal data related to the deceased person or pet. This personal data includes photos, videos, audio files, and text. The dedicated terminals then send this data to a server.

[1695] Data analysis and virtual entity generation

[1696] The server analyzes the received personal data. Image data is extracted using DeepFace to identify physical features of the deceased person or pet, while audio data is extracted using Google Text-to-Speech (gTTS) and other voice analysis tools. Text data is analyzed using the OpenAI API to identify frequently used words and speech patterns.

[1697] Based on these feature data, a generative AI model is used to generate virtual beings, which include 3D models and synthesized voices and are placed in a virtual space.

[1698] Providing a conversational interface

[1699] The dedicated terminal provides the user with a dialogue interface through which the user can interact with the virtual entity. Questions and comments entered by the user are sent to the server.

[1700] The server analyzes the user's input using a generative AI model and generates an appropriate response. The response is designed to be natural and takes into account the characteristics of the deceased person or pet. The generated response is sent to a dedicated terminal and displayed to the user.

[1701] Specific examples

[1702] For example, if a user uploads photos and videos of their beloved dog "Pochi" in a physical store and asks, "Pochi, how have you been lately?", the server analyzes this data and generates a virtual Pochi. Using a generative AI model, it generates a response to the question, "Pochi, how have you been lately?", returning the response, "I've been fine! How have you been?"

[1703] Prompt Sentence Examples

[1704] Below are some examples of prompt sentences.

[1705] Please reply as Pochi to the following questions: What have you been up to lately?

[1706] The system thus configured allows users to interact with virtual deceased loved ones or pets through natural conversations within a physical store, providing peace of mind.

[1707] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1708] Step 1:

[1709] Users use dedicated terminals (tablets, robots, or smart glasses) installed in physical stores to input personal data such as photos, videos, audio files, and text related to the deceased person or pet. The input data is sent to the server by the terminal. The input is completed when the user selects the data and presses the "upload" button.

[1710] Step 2:

[1711] The server stores the personal data received from the device and then begins analyzing it. Specifically, image data from photos and videos is used to extract physical features of the deceased person or pet using DeepFace. Voice data is used to extract features using Google Text-to-Speech (gTTS) and other voice analysis tools. Text data is analyzed using the OpenAI API to identify frequently used words and speaking patterns.

[1712] Step 3:

[1713] The server uses a generative AI model to generate a virtual presence (3D model and voice) based on the feature data extracted in step 2. This virtual presence reflects the unique characteristics of the deceased person or pet and is capable of natural conversation. The generated virtual presence is stored in a database for the virtual space.

[1714] Step 4:

[1715] The dedicated terminal provides a dialogue interface to the user. The user can use this interface to start a dialogue with the virtual entity. For example, when the user types "Pochi, how have you been lately?", the data is sent from the terminal to the server.

[1716] Step 5:

[1717] The server analyzes the user's questions and comments received in step 4 and generates an appropriate response using the generative AI model. This response is designed to be a natural conversation that takes into account the characteristics of the deceased person or pet. For example, in response to a user's question, "Pochi, how have you been lately?", the server generates a response such as, "I've been fine! How have you been?"

[1718] Step 6:

[1719] The generated response is sent from the server to a dedicated terminal and displayed to the user. The user can confirm the response and continue the dialogue. The content of the dialogue is recorded on the server and used as reference for future dialogues.

[1720] Step 7:

[1721] When the user finishes the conversation, the dedicated terminal closes the conversation interface and the server ends the session. If the user wishes, the conversation content can be saved and a link for the next access can be provided.

[1722] Through the above steps, users can have natural conversations with their deceased loved ones or pets in a physical store, providing peace of mind.

[1723] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1724] The present invention is a system that allows users to create virtual beings based on personal data related to deceased people or pets and engage in conversations in a virtual space. This system incorporates an emotion engine that analyzes the user's emotions, enabling more personalized conversations. The system operates primarily between a server, a terminal, and the user.

[1725] User Data Collection

[1726] 1. Data Collection

[1727] User: The user launches the app and selects personal data related to the deceased person or pet, including photos, videos, audio files, and text.

[1728] On the device: The device will preview the data you selected and ask you to confirm the upload. Once confirmed, the data will be uploaded to the server.

[1729] Server: The server stores the received data and notifies the user that it was received successfully.

[1730] Data analysis and generation

[1731] 2. Data Analysis

[1732] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[1733] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[1734] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[1735] Server: Stores these feature data in a database.

[1736] 3. Virtual Being Creation

[1737] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[1738] Server: Stores the generated virtual entities in a database for the virtual space.

[1739] Preparing for a virtual cemetery

[1740] 4. Resource Allocation and Link Creation

[1741] Server: Allocates resources for the virtual graveyard and places the generated virtual beings in the virtual space.

[1742] Server: Generates a link to access the virtual cemetery and sends it to the device.

[1743] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[1744] Dialogue and emotion engine usage

[1745] 5. Access and Authentication

[1746] User: Clicks on the provided access link to access the virtual cemetery.

[1747] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[1748] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[1749] 6. Start a dialogue

[1750] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[1751] Terminal: Sends user input to the server.

[1752] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[1753] Server: Sends the generated response to the terminal.

[1754] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[1755] Use of emotion engine

[1756] 7. Emotional Recognition

[1757] Device: The device is equipped with facial recognition and voice analysis functions, and analyzes the user's emotions in real time from their facial expressions and voice.

[1758] Server: Receives the analysis results and sends them to the emotion engine.

[1759] Server: The emotion engine adjusts the content and tone of the response based on the user's emotional data.

[1760] 8. Personalized responses

[1761] Server: Generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it will return words of encouragement.

[1762] Server: Sends the generated response to the terminal.

[1763] Terminal: Displays the response received from the server to the user, for example, "Cheer up, we're always thinking about you."

[1764] Specific examples

[1765] As an example, we will explain how a user can upload photos or videos of a deceased pet and start a dialogue using the emotion engine.

[1766] 1. Data Collection

[1767] User: Use the dedicated app to select and upload photos and videos of your pet.

[1768] On your device: Display the progress of the data upload and notify you when it's finished by saying "Upload complete."

[1769] Server: Stores the received data and prepares it for analysis.

[1770] 2. Data Analysis and Generation

[1771] Server: Extracts the pet's appearance characteristics from photos and videos and analyzes its voice characteristics.

[1772] Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[1773] 3. Preparing a Virtual Graveyard

[1774] Server: Allocates resources for the virtual cemetery and generates access links.

[1775] Terminal: Send the link to the user and tell them, "Your virtual cemetery is ready. Please access it from the following link: example_link."

[1776] 4. Use of dialogue and emotion engines

[1777] User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[1778] Terminal: Sends user input to the server.

[1779] Server: Analyzes the user's input and responds with "Pochi was fine! How was yours?"

[1780] Terminal: Displays the response from the server and shows it to the user.

[1781] Terminal: Analyzes emotions from the user's facial expressions and voice, and sends the results to the server.

[1782] Server: The emotion engine generates a response based on the user's emotions, for example, returning encouraging words such as "You're sad, we're thinking of you too."

[1783] The system allows users to virtually interact with deceased loved ones or pets and receive personalized responses based on their emotions, providing greater peace of mind.

[1784] The processing flow will be explained below.

[1785] Program processing steps

[1786] Step 1: Select and upload data

[1787] User: Launch the dedicated app and select photos, videos, audio, and text data of the deceased person or pet.

[1788] On the device: Preview the selected file and ask the user for confirmation. If the user presses the confirm button, the file will be uploaded to the server.

[1789] Server: Stores the received data in a database and notifies the device that the upload is complete.

[1790] Terminal: Notify the user that the upload was successful.

[1791] Step 2: Data analysis

[1792] Server: Analyzes photos and videos using image analysis algorithms to extract the external characteristics of the deceased person or pet.

[1793] Server: The voice data is analyzed using a voice analysis algorithm to extract voice characteristics and tones.

[1794] Server: Extracts and analyzes frequently used words and unique language patterns from text data.

[1795] Server: Stores these feature data in a database.

[1796] Step 3: Creating a Virtual Being

[1797] Server: Based on the analyzed feature data, a generative AI model is used to generate a virtual deceased person or pet. This generation process includes creating a 3D model and synthesizing voice.

[1798] Server: Stores the generated virtual entities in a database for the virtual space.

[1799] Step 4: Preparing the Virtual Graveyard and Creating Links

[1800] Server: Allocates resources for the virtual graveyard and places the generated virtual beings in the virtual space.

[1801] Server: Generates a link to access the virtual cemetery and sends it to the device.

[1802] Terminal: Notifies the user of the access link and informs them that the virtual cemetery is ready.

[1803] Step 5: Access and Authentication

[1804] User: Clicks on the provided access link to access the virtual cemetery.

[1805] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[1806] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[1807] Step 6: Start a conversation

[1808] User: Interact with deceased loved ones or pets in the virtual space. For example, by typing, "How have you been?"

[1809] Terminal: Sends user input to the server.

[1810] Server: Analyzes user input using an AI model and generates responses that are similar to what the deceased person or pet would give.

[1811] Server: Sends the generated response to the terminal.

[1812] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[1813] Step 7: Recognize emotions

[1814] Device: The device is equipped with facial recognition and voice analysis functions, and analyzes the user's emotions in real time from their facial expressions and voice.

[1815] Server: Receives the analysis results and sends them to the emotion engine.

[1816] Server: The emotion engine adjusts the content and tone of the response based on the user's emotional data.

[1817] Step 8: Personalize your response

[1818] Server: Generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it will return words of encouragement.

[1819] Server: Sends the generated response to the terminal.

[1820] Terminal: Displays the response received from the server to the user, for example, "Cheer up, we're always thinking about you."

[1821] Step 9: Additional Interactions

[1822] User: Continues to reminisce about and ask questions about deceased loved ones and pets. Performs specific actions in the virtual space, such as placing flowers.

[1823] Terminal: Sends user actions to the server and displays the results of the execution.

[1824] Server: Updates the virtual space in real time based on the user's actions, allowing the user to continue interacting.

[1825] In this way, users can virtually converse and interact with their deceased loved ones or pets, bringing them peace of mind.

[1826] Example 2

[1827] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1828] Today, many people use photos and videos to remember their deceased loved ones and pets, but these alone are not enough to bring them peace of mind. While there are systems that allow users to virtually interact with their deceased loved ones and pets, few systems can respond to emotional changes in real time. As a result, it is currently difficult for users to receive personalized experiences tailored to their emotions.

[1829] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to select multiple pieces of personal data related to the deceased person or pet; means for transmitting the selected personal data to the server; means for analyzing the personal data received by the server and extracting characteristics of the deceased person or pet; means for generating a virtual presence of the deceased person or pet using a generative AI model based on the extracted characteristics; means for placing the generated virtual presence in a virtual space; means for generating and providing a link for accessing the virtual space to the user; means for the user to interact with the virtual presence in the virtual space; an emotion engine for analyzing the user's emotions in real time; and means for personalizing responses based on the emotion engine. This allows the user to receive personalized responses according to their emotions while virtually interacting with the deceased person or pet.

[1830] "User" refers to an individual who uses this system.

[1831] "Personal data" refers to information such as photos, videos, audio files, and text related to the deceased person or pet.

[1832] "Terminal" refers to a device operated by a user, such as a computer, smartphone, or tablet.

[1833] "Server" refers to the computer that processes and stores data at the center of this system.

[1834] "Analysis" refers to the process of extracting characteristics and information about the deceased person or pet from personal data.

[1835] A "generative AI model" refers to an artificial intelligence model that uses machine learning algorithms to generate new data or formats.

[1836] "Virtual presence" refers to a digital representation of a deceased person or pet created by a generative AI model based on analyzed personal data.

[1837] "Virtual space" refers to a digital environment in which users interact with virtual beings.

[1838] "Link" refers to a URL or hyperlink for accessing a virtual space.

[1839] An "emotion engine" refers to a system that analyzes a user's emotions and generates a response based on those emotions.

[1840] The present invention provides a system that allows users to create virtual beings based on personal data related to deceased people or pets and to interact with them in a virtual space. This system is mainly composed of a server, terminals, and users.

[1841] User Data Collection

[1842] User: Launches the dedicated app and selects photos, videos, audio files, and text files related to the deceased person or pet.

[1843] On the device: Preview the selected data and ask the user to confirm the upload. Once confirmed, send the data to the server.

[1844] Server: Saves the received data in local storage or cloud storage and notifies the user that the upload is complete.

[1845] Data analysis and generation

[1846] Server: Analyzes the received photos and videos using image analysis algorithms (e.g., OpenCV or TensorFlow) to extract the external characteristics of the deceased person or pet.

[1847] Server: The audio file is analyzed using a speech analysis algorithm (e.g., Google Cloud Speech-to-Text or IBM Watson) to extract voice characteristics.

[1848] Server: Analyzes the text data using natural language processing algorithms (e.g., spaCy or BERT) to extract frequently used words and distinctive language patterns.

[1849] Server: Stores these feature data in a database (e.g., PostgreSQL or MongoDB).

[1850] Server: Based on the stored feature data, a generative AI model (e.g., GPT-3 or StyleGAN) is used to generate virtual deceased people or pets. This generation process includes creating 3D models and synthesizing voices. The generated virtual beings are stored in a database for the virtual space.

[1851] Preparing for a virtual cemetery

[1852] Server: Allocate resources for the virtual cemetery and place the virtual presence of deceased people and pets in the virtual space. This includes securing cloud server resources and generating the virtual space.

[1853] Server: Generates and provides users with a dedicated link to access the virtual cemetery.

[1854] Terminal: Notifies the user of the generated access link and informs them that the virtual cemetery is ready.

[1855] Use of dialogue and emotion engines

[1856] User: Clicks on the provided access link to access the virtual cemetery.

[1857] Terminal: Sends the user's authentication information to the server and executes the authentication process.

[1858] Server: If authentication is successful, the dialogue system is launched and the user is guided to the virtual cemetery.

[1859] User: Initiate a virtual conversation with a deceased loved one or pet, for example, by typing "How have you been?"

[1860] Terminal: Sends user input to the server.

[1861] Server: Analyzes the user's input using an AI model, generates responses that the deceased person or pet would likely make, and sends them to the device.

[1862] Terminal: Displays the response received from the server to the user, such as "I'm fine! How have you been?"

[1863] Use of emotion engine

[1864] Device: Using facial recognition and voice analysis functions, emotions are analyzed in real time from the user's facial expressions and voice.

[1865] Server: Receives the analysis results and sends them to the emotion engine.

[1866] Server: The emotion engine generates personalized responses by adjusting the content and tone of the response based on the user's emotional data. For example, if the user is sad, it adds words of encouragement.

[1867] Server: Sends the generated response to the terminal and displays it to the user, for example, "Cheer up, I'm always worried about you."

[1868] Specific examples

[1869] For example, consider a user uploading photos and videos of a deceased pet to use the system:

[1870] 1. User: Use the dedicated app to select and upload photos and videos of your pet.

[1871] 2. Device: Displays the progress of data upload and notifies you when it is complete.

[1872] 3. Server: Stores the received data and prepares it for analysis.

[1873] 4. Server: Extracts the pet's external features from photos and videos and analyzes its audio features.

[1874] 5. Server: Based on this data, a virtual pet is generated and placed in the virtual space.

[1875] 6. Server: Allocates resources for the virtual cemetery and generates access links.

[1876] 7. Terminal: Notify the user of the link and say, "Your virtual cemetery is ready. Please access it from the following link: example_link."

[1877] 8. User: Accesses the virtual graveyard and types, "Pochi, what have you been up to lately?"

[1878] 9. Terminal: Sends user input to the server and displays responses to the user. It displays responses such as "Pochi was fine! How was yours?"

[1879] 10. Terminal: Analyzes emotions from the user's facial expressions and voice and sends them to the server.

[1880] 11. Server: The emotion engine generates a response based on the user's emotions, returning encouraging words such as "You're sad, we're thinking of you too." This system allows the user to find greater peace of mind.

[1881] This Detailed Description provides a detailed understanding of how the present invention may be practiced.

[1882] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1883] Program processing steps

[1884] Step 1: Collect user data

[1885] Input: The user selects photos, videos, audio files, and text related to the deceased person or pet.

[1886] How it works: Launch the dedicated app and select data through the interface.

[1887] Output: The selected data is saved to the device and sent to the next processing step.

[1888] Step 2: Upload data

[1889] Input: Personal data stored on your device.

[1890] How it works: The device previews the selected data and asks the user to confirm the upload. Once confirmed, the data is sent to the server.

[1891] Output: Personal data is uploaded to the server.

[1892] Step 3: Data storage and notification

[1893] Input: Personal data received by the server.

[1894] How it works: The server saves the received data to local storage or cloud storage and notifies the user that the upload is complete.

[1895] Output: The data is saved and the user is notified that the upload is complete.

[1896] Step 4: Image analysis

[1897] Input: Photos and videos from your saved personal data.

[1898] How it works: The server uses image analysis algorithms (e.g., OpenCV or TensorFlow) to extract the external features of the deceased person or pet.

[1899] Output: Appearance feature data is generated.

[1900] Step 5: Audio analysis

[1901] Input: Audio file from saved personal data.

[1902] How it works: The server uses voice analysis algorithms (e.g., Google Cloud Speech-to-Text or IBM Watson) to extract voice characteristics and tone.

[1903] Output: Voice feature data is generated.

[1904] Step 6: Text Analysis

[1905] Input: A text file from your saved personal data.

[1906] How it works: The server uses natural language processing algorithms (e.g. spaCy or BERT) to analyze frequently used words and unique language patterns.

[1907] Output: Word and language pattern data is generated.

[1908] Step 7: Save to database

[1909] Input: Analyzed appearance feature data, voice feature data, and word and language pattern data.

[1910] How it works: The server stores this data in a database (e.g. PostgreSQL or MongoDB).

[1911] Output: Feature data stored in a database.

[1912] Step 8: Virtual Being Generation

[1913] Input: Stored feature data.

[1914] How it works: The server uses generative AI models (e.g., GPT-3 or StyleGAN) to generate virtual versions of deceased people and pets, including creating 3D models and synthesizing voices.

[1915] Output: Virtual entity data is generated.

[1916] Step 9: Virtual Space Placement

[1917] Input: Data of the generated virtual being.

[1918] How it works: The server allocates resources for the virtual cemetery and places virtual figures of deceased people and pets in the virtual space.

[1919] Output: A virtual presence of a deceased person or pet placed in a virtual space.

[1920] Step 10: Link generation and notification

[1921] Input: A virtual entity placed in a virtual space.

[1922] How it works: The server generates a dedicated link to access the virtual cemetery and provides it to the user.

[1923] Output: The generated access link is notified to the user.

[1924] Step 11: Authentication and Access

[1925] Input: The access link the user received.

[1926] How it works: The user clicks on the provided link to access the virtual cemetery. The device sends authentication information to the server, which then performs the authentication process.

[1927] Output: The user can now access the virtual graveyard.

[1928] Step 12: Start a conversation

[1929] Input: A prompt that the user types in the virtual space (e.g., "How have you been?").

[1930] How it works: The device sends user input to the server, which then analyzes the prompt and uses a generative AI model to generate a response for the deceased person or pet. The generated response is then sent to the device.

[1931] Output: The user will see a response from the deceased person or pet (e.g., "I'm fine! How have you been?").

[1932] Step 13: Emotion Recognition

[1933] Input: The user's facial expressions and voice during the conversation.

[1934] How it works: The device uses facial recognition and voice analysis to analyze the user's facial expressions and voice in real time.

[1935] Output: The parsed emotion data is generated.

[1936] Step 14: Personalize your response

[1937] Input: Parsed emotion data and the following user input:

[1938] Operation: The server uses the emotion engine to generate a response based on the user's emotion. The generated response is sent to the device.

[1939] Output: A personalized response is displayed to the user (e.g., "Cheer up, we're always worried about you").

[1940] This allows the user to enjoy a virtual conversation while receiving individual responses according to their emotions.

[1941] (Application example 2)

[1942] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1943] Conventional virtual dialogue systems have the problem that interactions with virtual beings generated based on personal data related to deceased loved ones or pets cannot flexibly respond to the user's emotions or circumstances, which limits the peace of mind that users can gain and makes it difficult to achieve personalized dialogue.

[1944] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for importing multiple pieces of personal data related to the deceased person or pet, means for analyzing the imported personal data and extracting characteristics of the deceased person or pet, means for generating a virtual presence of the deceased person or pet based on the extracted characteristics, means for providing a virtual space in which the user can interact with the generated virtual presence, means including an emotion engine for analyzing the user's emotions, and means for adjusting the content and tone of the response based on the analyzed emotion data. This enables personalized dialogue according to the user's emotions and situation.

[1945] "Deceased person or pet" refers to an object that the user recognizes as a deceased person or pet and recreates in the virtual space.

[1946] "Personal Data" refers to data containing personal information, such as photographs, videos, audio files, and text related to a deceased person or pet.

[1947] "Means for extracting features" refers to the processes and techniques for analyzing and extracting the appearance, vocal characteristics, and language patterns of the deceased person or pet from the collected personal data.

[1948] "Means for generating virtual entities" refers to the technology and process for generating virtual entities, such as 3D models and voices of deceased individuals or pets, based on extracted feature data.

[1949] "Virtual space" refers to a digitally simulated environment in which users can interact with generated virtual beings.

[1950] An "emotion engine" refers to software and algorithms that analyze a user's emotions from their facial expressions and voice and adjust the content and tone of their responses accordingly.

[1951] "Means for providing an interactive virtual space" refers to the technologies and processes for constructing and making accessible a virtual space that allows users to interact with generated virtual beings.

[1952] "Emotion data" refers to information about emotions obtained by analyzing the user's facial expressions and voice.

[1953] "Response content" refers to the content of a reply to an input made by a user to a virtual entity.

[1954] "Tone of response" refers to the nuances and tone of voice of the response to the user that are adjusted by the emotion engine.

[1955] An embodiment of the present invention will be described in detail below. The present invention is a system that allows a user to generate a virtual entity based on personal data related to a deceased person or pet and to engage in a dialogue in a virtual space. The system incorporates an emotion engine and includes a function for analyzing the user's emotions and providing a more personalized dialogue.

[1956] Users use a dedicated application to collect personal data related to the deceased person or pet. This application is installed on a smartphone and allows users to upload data such as photos, videos, audio files, and text. The device previews the data selected by the user, asks for confirmation of the upload, and once confirmation is received, sends the data to the server. The server stores the received data and notifies the user that it has been successfully received.

[1957] Using the generative AI model, the server analyzes the received data and extracts the characteristics of the deceased person or pet. Specifically, it uses an image analysis algorithm to extract external features from photos and videos, and an audio analysis algorithm to extract voice characteristics and tone from audio data. It also analyzes frequently used words and unique language patterns from text data and stores this feature data in a database. Based on the analyzed feature data, the server creates a 3D model and synthesizes voice to generate a virtual deceased person or pet.

[1958] The generated virtual beings are placed in a virtual space. The server allocates resources for the virtual space, places the generated virtual beings in the virtual space, and generates a link to access the virtual cemetery. This link is provided to the user via the terminal, and the user clicks on the provided link to access the virtual cemetery. The terminal sends the user's authentication information to the server, and if the authentication is successful, a dialogue system is launched and guides the user to the virtual cemetery.

[1959] Users can interact with their deceased loved ones or pets in a virtual space. For example, by inputting a question such as "How have you been?", the device sends the input to the server, which then analyzes it using a generative AI model to generate a response that the deceased or pet would likely give. The generated response is then sent to the device and displayed to the user.

[1960] The device also provides a function that analyzes the user's emotions in real time using an emotion engine. The device is equipped with facial recognition and voice analysis functions, which analyze the user's emotions from their facial expressions and voice and send the results to the server. The server then uses the emotion engine to adjust the content and tone of the response based on the analysis results, generating a personalized response that matches the user's emotions.

[1961] Here are some examples of prompts:

[1962] User: Pochi, what have you been up to lately?

[1963] System: Pochi was fine! How were you?

[1964] The system allows users to virtually interact with deceased loved ones or pets and receive personalized responses based on their emotions, providing greater peace of mind.

[1965] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1966] Step 1:

[1967] The user launches a dedicated application and selects personal data (photos, videos, audio files, text, etc.) related to the deceased person or pet. The application prompts the user to upload the selected data. The application previews the data selected by the user and asks for confirmation of the upload. Once the user confirms, the device sends the data to the server. The input is the personal data selected by the user, and the output is the transmission of the data to the server.

[1968] Step 2:

[1969] The server stores the received personal data and notifies the user that it has been received successfully. The server then uses image analysis algorithms to analyze the photos and videos to extract the physical characteristics of the deceased person or pet. It also uses voice analysis algorithms to analyze the audio data to extract voice characteristics and tone. It also extracts frequently used words and distinctive language patterns from the text data. The input is the personal data uploaded to the server, and the output is data on physical characteristics, voice characteristics, and language patterns.

[1970] Step 3:

[1971] The server uses a generative AI model to generate a virtual deceased person or pet based on the analyzed feature data. This generation process includes creating a 3D model and synthesizing voice. The data of the generated virtual being is stored in a database for the virtual space. The input is the extracted feature data, and the output is the data of the generated virtual being.

[1972] Step 4:

[1973] The server allocates resources for the virtual cemetery and places the generated virtual beings in the virtual space. Next, it generates a link to access the virtual cemetery and sends it to the terminal. The terminal notifies the user of this link and informs them that the virtual cemetery is ready. The input is the data of the virtual beings, and the output is the generation and notification of the access link.

[1974] Step 5:

[1975] The user clicks on the provided access link to access the virtual cemetery. The terminal sends the user's authentication information to the server and executes the authentication process. If the authentication is successful, the server launches a dialogue system and guides the user to the virtual cemetery. The input is the user's access link and authentication information, and the output is access to the virtual cemetery and the launch of the dialogue system.

[1976] Step 6:

[1977] The user interacts with the deceased person or pet in the virtual space. For example, they might type, "How have you been?" The device sends the user's input to the server. The server analyzes the user's input using a generative AI model and generates a response that the deceased person or pet would likely give. The generated response is sent to the device and displayed to the user. The input is the user's interaction, and the output is the generated response.

[1978] Step 7:

[1979] The device uses facial recognition and voice analysis functions to analyze emotions from the user's facial expressions and voice. The analysis results are sent to the server. The server's emotion engine adjusts the response content and tone based on the user's emotional data. The input is the user's facial expressions and voice data, and the output is the adjusted response content and tone.

[1980] Step 8:

[1981] The server generates an appropriate response based on the analysis results of the emotion engine. For example, if the user is sad, it generates words of encouragement and sends the generated response to the device, which then displays it to the user. The input is the analysis results of the emotion engine, and the output is a personalized response.

[1982] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1983] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1984] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1985] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1986] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1987] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1988] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1989] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1990] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1991] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1992] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1993] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1994] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1995] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1996] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1997] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1998] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1999] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2000] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2001] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope ...

Claims

1. a means for capturing a plurality of personal data relating to the deceased person or pet; A means for analyzing the captured personal data and extracting characteristics of the deceased person or pet; means for generating a virtual presence of the deceased person or pet based on the extracted features; means for providing a virtual space in which a user can interact with the generated virtual being; A system including:

2. 2. The system according to claim 1, further comprising: means for inputting personal data from a user terminal; and means for transmitting data to the server.

3. The system according to claim 1 , further comprising means for generating and providing a link for accessing said virtual space to a user.

Images