system

Abstract

To provide an effective hologram generating system for easily allowing bereaved families to remember the deceased based on the data of the deceased. [Solution] A system including: means for uploading data such as the deceased's audio, photos, videos, social media posts, words and actions, essays, letters, etc.; means for verifying the consistency and completeness of the uploaded data; means for analyzing text data using natural language processing technology and extracting characteristics of the deceased's words and actions; means for analyzing audio data using voice recognition technology and extracting the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology and extracting the deceased's appearance and movements; means for generating a hologram and voice model based on the generated characteristics of the deceased; means for transferring the hologram and voice model to a hologram device; and means for displaying a hologram of the deceased and allowing a user to interact with the deceased.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, when a family member or loved one passes away, it is difficult for surviving family members to find an appropriate way to heal their grief. While there are technologies available to recreate the voice and appearance of the deceased, these methods are complex, and it is difficult to integrate individual data to create natural-looking holograms. This invention solves this problem by providing an effective hologram generation system based on the deceased's data, allowing surviving family members to easily remember the deceased. [Means for solving the problem]

[0005] The present invention provides a system for uploading data such as the deceased's voice, photos, videos, social media posts, words, actions, essays, and letters, and a system for verifying the integrity and completeness of the uploaded data. It also provides a system for analyzing text data using natural language processing technology to extract characteristics of the deceased's words and actions, a system for analyzing voice data using speech recognition technology to extract the deceased's voice quality and speaking style, and a system for analyzing photos and videos using image recognition technology to extract the deceased's appearance and movements. This system includes a system for generating a hologram and a voice model based on the generated characteristics of the deceased. The generated hologram and voice model are transferred to a hologram device, allowing users to enjoy interacting with the deceased. This system allows bereaved families to relive precious memories of the deceased and find emotional solace.

[0006] "Data of the deceased" refers to information such as audio, photographs, videos, social media posts, words and actions, essays, letters, etc. left behind by the deceased during their lifetime.

[0007] "Means of uploading" refers to the interface or process by which a user submits data about a deceased person to the system.

[0008] "Integrity" refers to the consistency of data format and content according to specified standards.

[0009] "Integrity" refers to the completeness of data, with no missing or corrupted data.

[0010] "Natural language processing technology" refers to technology that enables computers to understand, analyze, and generate human language.

[0011] "Voice recognition technology" refers to the technology that analyzes voice data, converts it into text data, and extracts its features.

[0012] "Image recognition technology" refers to the technology of analyzing visual data such as photographs and videos and extracting specific information and features.

[0013] A "3D model" refers to a three-dimensional object generated on a computer.

[0014] "Voice model" refers to a digital audio file created to recreate the voice of a deceased person.

[0015] "Hologram" refers to a technology that uses the interference effect of light to reproduce three-dimensional images.

[0016] "Hologram device" refers to a hardware device for displaying a hologram.

[0017] An "interface" refers to the point of contact through which humans interact with computers or systems.

[0018] "Users" refers to the bereaved families and related parties who use the system. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system that uses generative AI to create holograms based on data from the deceased's life, providing comfort to surviving family members and loved ones. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[0041] Uploading data

[0042] The user accesses the system and uploads the necessary data on the deceased.

[0043] Example: A user accesses the system on their home computer or smartphone and uploads photos of their deceased grandfather and past messages.

[0044] Receipt and confirmation of data

[0045] The server receives the uploaded data and verifies its consistency and completeness.

[0046] For example: A server might validate uploaded file types and sizes, ensure images load properly, and ensure audio files aren't corrupted.

[0047] Data analysis

[0048] The server analyzes the received data and extracts the necessary information.

[0049] Using natural language processing (NLP) technology, the characteristics of the deceased's words and actions are analyzed from text data.

[0050] Using voice recognition technology, characteristic voice qualities and speaking styles are analyzed from audio data.

[0051] Image recognition technology is used to analyze the face and gestures of the deceased from photographs and videos.

[0052] Example: The server understands the context of messages and letters from the deceased and identifies common phrases and speaking characteristics.

[0053] Hologram generation

[0054] The generating AI generates a hologram of the deceased person based on the analyzed data.

[0055] Using 3D modeling technology, a 3D model is created that recreates the appearance of the deceased.

[0056] Using voice synthesis technology, an audio file is created that reproduces the voice of the deceased.

[0057] Motion capture technology is used to create animations that recreate the movements of the deceased.

[0058] Example: Generative AI creates a 3D model of your grandfather based on old photos and generates natural-sounding conversational voices based on audio recordings.

[0059] Hologram data transfer

[0060] The server transfers the generated hologram data to the hologram device.

[0061] The generated 3D model, audio file, and motion data are packaged into a single file and transferred to the hologram device.

[0062] Example: The server notifies the user that "the hologram of your grandfather is complete" and transfers the data to the device.

[0063] Hologram display

[0064] The user activates the hologram device, which displays a hologram of the deceased person.

[0065] Turn on the hologram device and start the specified application.

[0066] Example: A user turns on a hologram device in their living room and relives a holographic story their grandfather used to tell their family.

[0067] Interaction and Continued Use

[0068] Users enjoy interacting with the holograms.

[0069] Users can talk to the hologram and carry on a conversation.

[0070] New data analyses and topics are added to the hologram periodically.

[0071] Example: A user can ask a hologram, "What story would you like to tell us today?" and the hologram will relive a story from an old family photo.

[0072] As a result, this system can effectively utilize data from the deceased's lifetime, providing deep healing for family members and those involved.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] Users access the system and upload data about the deceased. Specifically, they log in and open the data upload page. From there, they select files such as photos, audio, video, social media posts, essays, and letters, and press the upload button to send them to the server.

[0076] Step 2:

[0077] The server receives the uploaded data. Specifically, the server receives the data file sent by the user and saves it in a specified storage location.

[0078] Step 3:

[0079] The server checks the integrity and completeness of the data it receives. Specifically, it checks the file format, size, and data corruption status to check for any abnormalities. If an abnormality is found, it sends a notification to the user urging them to re-upload.

[0080] Step 4:

[0081] The server analyzes the data. Specifically, it uses natural language processing (NLP) technology to extract characteristics of the deceased's words and actions from the text data. It also uses a language model to analyze frequently occurring phrases and context.

[0082] Step 5:

[0083] The server uses voice recognition technology to analyze the audio data, extracting characteristics of the deceased's voice quality and speaking style from the audio data and storing these characteristics in a database.

[0084] Step 6:

[0085] The server uses image recognition technology to analyze photos and videos, extracting facial features and gestures of the deceased, and gathers the information needed to generate a 3D model.

[0086] Step 7:

[0087] Based on the analyzed data, the AI ​​generates a hologram of the deceased. Specifically, it uses 3D modeling technology to create a 3D model that recreates the appearance of the deceased, voice synthesis technology to generate an audio file that recreates the deceased's voice, and motion capture technology to create an animation that recreates the movements of the deceased.

[0088] Step 8:

[0089] The server transfers the generated hologram data to the hologram device. Specifically, it packages the generated 3D model, audio file, and motion data, and transfers the package to the hologram device.

[0090] Step 9:

[0091] The server notifies the user that the hologram data transfer has been completed. Specifically, the server notifies the user by sending a pop-up screen or email to the user's device stating that "the hologram data has been transferred to the device."

[0092] Step 10:

[0093] The user activates the hologram device and displays the hologram. Specifically, the user turns on the device, launches the application, and displays the transferred hologram of the deceased person.

[0094] Step 11:

[0095] Users can enjoy interacting with the hologram by speaking to it or giving it specific instructions, thereby recreating memories of the deceased.

[0096] Through the above processing steps, a hologram based on the data of the deceased person is generated, and the user can find emotional healing through it.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] In modern society, people deeply cherish memories of their deceased loved ones, but there are limited ways to physically experience them. Traditional memorial methods struggle to recreate the voices and movements of the deceased, resulting in the bereaved family members being unable to feel close to them. In particular, there is a need for a method to heal the hearts of bereaved family members by recreating the voices and movements of the deceased.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes: means for a user to upload data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for the server to verify the consistency and completeness of the uploaded data; means for the server to analyze the text data using natural language processing technology and extract characteristics of the deceased's words and actions; means for the server to analyze the voice data using voice recognition technology and extract the deceased's voice quality and speaking style; means for the server to analyze photos and videos using image recognition technology and extract the deceased's appearance and movements; means for the generation AI to generate a hologram and a voice model based on the generated characteristics of the deceased; means for the server to transfer the hologram and voice model to a hologram device; and means for the user to display a hologram of the deceased and interact with the deceased. This allows the user to interact with the hologram of the deceased and experience a physical sensation of the deceased.

[0102] "User" refers to an individual who accesses the system, uploads data of a deceased person, and uses a hologram device.

[0103] "Server" refers to the device or system that receives, verifies, and analyzes data uploaded by users, and generates and transmits holograms and audio models.

[0104] A "deceased person" refers to a person who has already passed away, but about whom the user wishes to retain memories.

[0105] "Audio data" refers to files containing recordings of the deceased person's voice.

[0106] "Photograph" refers to a still image that visually records the appearance of the deceased.

[0107] "Video" refers to a dynamic image file that includes the movements of the deceased.

[0108] "Social media posts" refers to data such as text, images, and videos posted by the deceased on social media platforms on the Internet.

[0109] "Words and actions" refers to the words and actions of the deceased.

[0110] "Composition" refers to a document written by the deceased.

[0111] A "letter" refers to a document written by a deceased person as a form of personal communication.

[0112] "Data Integrity" refers to the state in which uploaded data is accurate and complete.

[0113] "Natural language processing technology" refers to the technology of analyzing text data and extracting meaning from it.

[0114] "Voice recognition technology" refers to technology that analyzes voice data and extracts features from the voice.

[0115] "Image recognition technology" refers to the technology of analyzing images and videos and extracting features from them.

[0116] "Generative AI" refers to artificial intelligence technology that automatically generates holograms and audio models based on analyzed data.

[0117] "Hologram device" refers to a device that displays generated holograms and allows a user to interact with them.

[0118] "Interaction" refers to a two-way exchange between a user and a hologram.

[0119] The present invention is a system that uses AI to create holograms based on data from the deceased person's life, providing comfort to surviving family members and those involved. Specific embodiments for carrying out the present invention are described below.

[0120] Uploading data

[0121] The user accesses the system and uploads the necessary data about the deceased. This is accomplished by logging into the system's web portal using a home computer or smartphone and uploading photos, audio files, videos, text files, etc. of the deceased. For example, a user can select photos of their deceased grandfather and past messages to be imported into the system.

[0122] Receipt and confirmation of data

[0123] The server receives the uploaded data and checks its consistency and integrity. Specifically, it checks the file format (e.g., jpg, mp3, mp4, txt, etc.) of each file received by the server, and then calculates the file's hash value to ensure the data is not corrupted. If an improper file format or corruption is detected, an error message is returned to the user.

[0124] Data analysis

[0125] The server analyzes the received data and extracts the necessary information. Natural language processing (NLP) technology is used to analyze the text data and extract characteristics of the deceased's words and actions. For example, distinctive phrases and speaking characteristics are extracted from letters and emails uploaded by the user. Speech recognition technology is also used to analyze the audio data and identify the deceased's voice quality and intonation. Image recognition technology is used to analyze the deceased's face and movements from photos and videos. Specific software used at this stage is Python libraries (NLTK, SpaCy, OpenCV, etc.) and machine learning frameworks such as TENSORFLOW (registered trademark).

[0126] Hologram generation

[0127] The generative AI generates a hologram of the deceased based on the analyzed data. 3D modeling technology is used to create a 3D model that replicates the appearance of the deceased. Voice synthesis technology is used to create an audio file that replicates the deceased's voice. Motion capture technology is then used to create an animation that replicates the natural movements of the deceased. Specific software used at this stage includes Blender (3D modeling) and Vosk (voice synthesis). The prompt is something like, "Generate a 3D hologram of my grandfather based on his old photos and a recorded message."

[0128] Hologram data transfer

[0129] The server transfers the generated hologram data to the hologram device. The generated 3D model, audio file, and motion data are packaged and transferred to the hologram device as a single file. Once the transfer is complete, the user is notified.

[0130] Hologram display and interaction

[0131] The user activates the hologram device, which displays a hologram of the deceased. The user turns on the device and launches a specified application. The hologram device reads the data transferred from the server and displays a 3D hologram of the deceased. It plays audio and runs animations in real time. The user can talk to the hologram, which responds by reflecting the characteristics of the deceased. Enjoying this interaction brings back fond memories of the deceased and brings emotional healing.

[0132] In this way, the invention utilizes data from the deceased person's lifetime to provide a hologram that is extremely valuable to their family and friends.

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] Step 1:

[0135] Users access the system and upload data such as audio, photos, videos, social media posts, words, actions, essays, and letters of the deceased. The input is various data files (e.g., jpg, mp3, mp4, txt, etc.) owned by the user, and the output is the data files imported into the system. Specifically, users log in to the system's web portal using their home computer or smartphone and perform the upload operation.

[0136] Step 2:

[0137] The server receives data uploaded by users and checks its integrity and completeness. The input is the data file uploaded by the user, and the output is the data file whose integrity and completeness have been checked. Specifically, the server verifies the format of each file (e.g., jpg, mp3, mp4, etc.) and calculates a hash value to ensure the file is not corrupted. If the file is corrupted or in an improper format, an error message is returned to the user.

[0138] Step 3:

[0139] The server analyzes the received data and extracts the necessary information. The input is a data file whose consistency and completeness has been confirmed, and the output is a dataset with extracted characteristics of the deceased. Specific operations include using NLP technology to analyze the text data and extract commonly used phrases and speaking characteristics; analyzing the audio data using speech recognition technology to identify the voice quality and intonation of the deceased; and using image recognition technology to extract the face and movements of the deceased from photos and videos. Specific software used is Python libraries (NLTK, SpaCy, OpenCV, etc.) and TensorFlow.

[0140] Step 4:

[0141] The generation AI generates a hologram of the deceased based on the analyzed data. The input is the characteristic data of the deceased analyzed on the server, and the output is a hologram file containing a 3D model, audio file, and motion data. Specific operations involve using 3D modeling technology to create a 3D model that recreates the appearance of the deceased, and using voice synthesis technology to generate an audio file that recreates the deceased's voice. Furthermore, motion capture technology is used to create an animation that recreates the movements of the deceased. The specific software used is Blender (3D modeling) and Vosk (voice synthesis).

[0142] Step 5:

[0143] The server transfers the generated hologram data to the hologram device. The input is the generated hologram file, and the output is the data stored in the hologram device. Specifically, the server packages the generated 3D model, audio file, and motion data and transfers it to the hologram device as a single file. Once the transfer is complete, the server notifies the user that the data transfer is complete.

[0144] Step 6:

[0145] The user starts the hologram device and displays a hologram of the deceased. The input is a hologram file stored in the hologram device, and the output is a hologram of the deceased that is displayed and played back. Specifically, the user turns on the hologram device and launches a specified application. The hologram device reads the data, displays a 3D hologram of the deceased, plays audio in real time, and runs animations.

[0146] Step 7:

[0147] Users can enjoy interacting with the hologram. The input is responses from the hologram device and user input (voice and gestures), and the output is an interactive exchange experience. Specifically, when the user speaks to the hologram, the hologram responds using pre-analyzed data and real-time voice recognition technology. Continuing conversation and interaction with the user can revive fond memories of the deceased, bringing emotional healing.

[0148] (Application example 1)

[0149] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0150] Systems that more realistically recreate memories of the deceased and provide comfort to surviving family and friends face many technical challenges. In particular, systems that not only accurately reflect the characteristics of the deceased but also deepen communication with users through real interactions are required. Another challenge is to personalize the shopping experience and provide special value by recommending products based on the user's purchasing preferences.

[0151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0152] In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing the text data using natural language processing technology to extract characteristics of the deceased's words and actions; means for analyzing the voice data using voice recognition technology to extract the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology to extract the deceased's appearance and movements; means for generating a hologram and a voice model based on the generated characteristics of the deceased; means for transferring the hologram and the voice model to a hologram device; means for displaying a hologram of the deceased and allowing the user to interact with the deceased; and means for recommending products based on the user's purchasing preferences. This enables the user's purchasing experience to be personalized while realistically recreating the deceased's memories.

[0153] A "deceased person" refers to a person who has passed away and is the person about whom data is to be reproduced.

[0154] "Data upload" refers to a user sending information about a deceased person (such as audio, photos, videos, social media posts, words, actions, writings, letters, etc.) to the system.

[0155] "Data integrity and completeness" means ensuring that uploaded data is accurate, complete, and not corrupted or missing.

[0156] "Natural language processing technology" is a technology that analyzes text data and extracts the characteristics and meaning of sentences and behavior.

[0157] "Voice recognition technology" is a technology that analyzes voice data and extracts characteristics such as speaking style and voice quality.

[0158] "Image recognition technology" is a technology that analyzes photographic and video data to extract a person's appearance and movements.

[0159] A "hologram" is a three-dimensional image reproduced to recreate the appearance and movements of a deceased person.

[0160] A "voice model" is digital voice data generated to reproduce the voice of a deceased person.

[0161] A "hologram device" is a hardware device for displaying generated holograms and audio models.

[0162] "Interaction" means that the user can have a conversation or dynamic interaction with the hologram of the deceased person.

[0163] "Purchasing preferences" refers to purchasing tendencies based on a user's interests and preferences.

[0164] "Product recommendation" refers to the process of presenting relevant products based on a user's purchasing preferences.

[0165] The system for implementing this invention generates a hologram and a voice model using various data of the deceased person, and allows the user to interact with the hologram and voice model and receive personalized product recommendations. Specific embodiments are described below.

[0166] System Overview

[0167] The system consists of the following main components:

[0168] 1. Data upload method

[0169] 2. Means of verifying data consistency and completeness

[0170] 3. Data analysis methods (natural language processing technology, voice recognition technology, image recognition technology)

[0171] 4. Hologram and audio model generation method

[0172] 5. Transfer method to hologram device

[0173] 6. Holographic Display and Interaction Methods

[0174] 7. Product recommendation means

[0175] Hardware and software used

[0176] Hardware:

[0177] Common devices such as smartphones, tablets, and personal computers

[0178] Dedicated hologram device

[0179] software:

[0180] Server Application

[0181] Natural Language Processing (NLP) Engine

[0182] Speech Recognition Engine

[0183] Image Recognition Engine

[0184] Generative AI Models

[0185] Speech synthesis engine

[0186] Product recommendation engine

[0187] Operation flow

[0188] 1. Data upload:

[0189] Users use their own devices (such as smartphones) to upload data such as audio, photos, videos, social media posts, essays, and letters of the deceased to the system.

[0190] 2. Data consistency and integrity checks:

[0191] The server checks the integrity and completeness of the uploaded data, specifically by validating the file format, size, and content, and prompting the user to re-upload if there are any issues.

[0192] 3. Data Analysis:

[0193] The server will analyze the uploaded data using the following analysis techniques:

[0194] Text analysis: Using NLP techniques to extract characteristics of the deceased's words and actions from text data.

[0195] Speech analysis: Using speech recognition technology to extract voice quality and speaking style from audio data.

[0196] Image analysis: Using image recognition technology to analyze the appearance and movements of the deceased from photographs and videos.

[0197] 4. Hologram and audio model generation:

[0198] A generative AI model is used to generate a hologram and voice model of the deceased person based on the acquired feature data.

[0199] 5. Transfer to hologram device:

[0200] The generated hologram and audio model are packaged and transferred to the hologram device.

[0201] 6. Hologram Display and Interaction:

[0202] The user activates the hologram device and interacts with the hologram of the deceased person. The hologram responds to the user's input, realizing a dialogue with the user.

[0203] 7. Product recommendation:

[0204] Using a product recommendation engine, the hologram suggests products based on the user's purchasing preferences, just like a shop assistant, and explains the reasons and features of the recommended products.

[0205] Prompt Sentence Examples

[0206] Examples:

[0207] User: The user opens a shopping app on their smartphone and enjoys shopping while viewing a hologram of their deceased mother. When the user says, "I want a new shirt," the hologram suggests several shirts and explains why it recommends them.

[0208] Example prompt sentence:

[0209] text

[0210] Show me a hologram of a deceased person who knows my preferences and suggests products in a specific category (e.g., "shirts"), how I chose them, and why I recommend them.

[0211] Using this system, users can not only recreate memories of their deceased loved ones, but also have a personalized shopping experience through interactions with the deceased.

[0212] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0213] Step 1:

[0214] A user uses a device (smartphone or tablet) to upload data about the deceased (audio, photos, videos, social media posts, words, actions, essays, letters, etc.) to the system. These files are sent to the server as input data. The server temporarily stores the received data as output.

[0215] Step 2:

[0216] The server verifies the integrity and completeness of the uploaded data. As input, it validates the uploaded data file, checking the file format, size, whether it is corrupted, and ensuring that all data has been received correctly. As output, it generates a flag indicating whether the data is valid.

[0217] Step 3:

[0218] The server analyzes the received data. The input data is the deceased person's data received in step 1. Specifically, the following process is performed:

[0219] Using natural language processing technology, characteristics of the deceased's words and actions are extracted from text data.

[0220] Using voice recognition technology, voice quality and speaking style are extracted from audio data.

[0221] Using image recognition technology, the appearance and movements of the deceased are extracted from photographs and videos.

[0222] As an output, feature data of the deceased extracted by each technique is generated.

[0223] Step 4:

[0224] A generative AI model is used to generate a hologram and voice model based on the analyzed feature data. The feature data obtained in step 3 is used as input. Specifically, the appearance of the deceased is recreated using 3D modeling software, and the voice is recreated using voice synthesis technology. The output is hologram data and voice data.

[0225] Step 5:

[0226] The server transfers the generated hologram and audio data to the hologram device. As input, the data generated in step 4 is used. As output, it is sent to the hologram device, where it is converted into a displayable format.

[0227] Step 6:

[0228] The user activates the hologram device and displays a hologram of the deceased person. As input, hologram and audio data are loaded into the hologram device. Specifically, the user turns on the device and launches a dedicated application. As output, a hologram of the deceased person is displayed and interaction begins.

[0229] Step 7:

[0230] The hologram recommends products based on the user's purchasing preferences. The user's purchasing history and preference data are used as input. Specifically, the product recommendation engine analyzes the user's preferences and generates a product list based on them. As output, the hologram explains to the user the recommended products, their features, and the reasons for the recommendation.

[0231] Prompt Sentence Examples

[0232] Show me a hologram of a deceased person who knows my preferences and suggests products in a specific category (e.g., "shirts"), how I chose them, and why I recommend them.

[0233] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0234] This invention is a system that uses generative AI to create a hologram based on data from the deceased's life, and combines it with an emotion engine to respond according to the user's emotional state. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[0235] Uploading data

[0236] A user accesses the system and uploads data about the deceased. For example, a user accesses the system on a home computer or smartphone and uploads photos of a deceased grandfather or past messages.

[0237] Receipt and confirmation of data

[0238] The server receives the uploaded data and checks its consistency and completeness. For example, the server validates the uploaded file format and size, ensures that images load properly, and that audio files are not corrupted.

[0239] Data analysis

[0240] The server analyzes the received data and extracts the necessary information.

[0241] Using natural language processing (NLP) technology, the characteristics of the deceased's words and actions are analyzed from text data.

[0242] Using voice recognition technology, characteristic voice qualities and speaking styles are analyzed from audio data.

[0243] Using image recognition technology, the server analyzes the face and body language of the deceased from photos and videos. For example, the server can understand the context of messages and letters from the deceased and identify common phrases and speaking characteristics.

[0244] Hologram generation

[0245] The generating AI generates a hologram of the deceased person based on the analyzed data.

[0246] 3D modeling technology is used to create a 3D model that recreates the appearance of the deceased.

[0247] Voice synthesis technology is used to create an audio file that reproduces the voice of the deceased.

[0248] Motion capture technology is used to create animations that replicate the movements of the deceased, while generative AI creates a 3D model of your grandfather based on old photos, and generates natural-sounding conversational voices based on audio recordings.

[0249] Hologram data transfer

[0250] The server transfers the generated hologram data to the hologram device, packages the generated 3D model, audio file, and motion data, and transfers the package to the hologram device.

[0251] User Notifications

[0252] The server notifies the user that the hologram data transfer has been completed. For example, the server notifies the user by sending a message to the user's device via a pop-up screen or email saying "The hologram data has been transferred to your device."

[0253] Hologram display

[0254] The user activates the hologram device and displays the hologram of the deceased person, e.g., by turning on the device, launching an application, and displaying the transferred hologram of the deceased person.

[0255] Interaction and Emotion Recognition

[0256] Users can enjoy interacting with the holograms. By speaking to the holograms or giving them specific instructions, users can interact with the holograms and relive memories of their deceased loved ones.

[0257] Use of emotion engine

[0258] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness."

[0259] The hologram's behavior and responses are tailored based on the identified emotional state: for example, if the user is sad, the hologram will offer a comforting message or a reminiscence.

[0260] The emotion engine has a means to play pre-set memories and messages according to a specific emotional state. For example, if the user is emotional, the hologram will convey a warm message according to that emotion.

[0261] As a result, this system effectively utilizes data from the deceased person's lifetime and provides optimal responses according to the user's emotional state, thereby achieving deep healing.

[0262] The processing flow will be explained below.

[0263] Step 1:

[0264] Users access the system and upload data about the deceased. For example, they log in to a dedicated web application on their home computer or smartphone, select files such as photos, audio, video, social media posts, essays, and letters of the deceased, and press the upload button to send them to the server.

[0265] Step 2:

[0266] The server receives the uploaded data and checks its integrity and completeness. Specifically, it checks the file format and size of the data received, ensures that images are loaded properly, and that audio files are not corrupted. If there are any abnormalities, it sends a notification to the user, encouraging them to re-upload.

[0267] Step 3:

[0268] The server analyzes the received data. Specifically,

[0269] Using natural language processing (NLP) technology, characteristics of the deceased's words and actions are extracted from text data (letters, social media posts, etc.).

[0270] Voice recognition technology is used to extract the voice quality and speaking style of the deceased from audio data (such as recordings).

[0271] Image recognition technology is used to extract facial and movement characteristics of the deceased from photographs and videos.

[0272] Step 4:

[0273] The AI ​​generates a hologram of the deceased person based on the analyzed data.

[0274] Using 3D modeling technology, a 3D model is created that recreates the appearance of the deceased based on collected facial and body features.

[0275] Using voice synthesis technology, an audio file is created that reproduces the voice of the deceased based on the extracted voice quality and speaking characteristics.

[0276] Motion capture technology is used to create animations that recreate the movements of the deceased.

[0277] Step 5:

[0278] The server transfers the generated hologram data to the hologram device. Specifically, it combines the generated 3D model, audio file, and motion data into a single package and sends it to the hologram device.

[0279] Step 6:

[0280] The server notifies the user that the hologram data transfer has been completed. Specifically, the server sends a pop-up notification or email to the user's device stating, "Hologram data has been transferred to your device."

[0281] Step 7:

[0282] The user activates the hologram device and displays the hologram of the deceased. Specifically, the user turns on the hologram device, launches the dedicated application, reads the transferred hologram data, and displays the hologram of the deceased.

[0283] Step 8:

[0284] Users can enjoy interacting with holograms. By talking to the hologram or giving specific instructions, the hologram will respond, establishing a dialogue.

[0285] Step 9:

[0286] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. Specifically, the hologram device captures the user's facial expressions and tone of voice through a built-in camera and microphone, and the emotion engine analyzes the data to determine emotions such as "sadness," "joy," and "surprise."

[0287] Step 10:

[0288] The server adjusts the hologram's behavior and responses based on the identified emotional state. For example, if the user is sad, the hologram will offer comforting words or warm messages, including reminiscences. If the user is happy, the hologram will engage in proactive interactions tailored to that emotion.

[0289] Step 11:

[0290] The server plays pre-set memories and messages according to the emotional state identified by the emotion engine. For example, if the user is emotional, a special message corresponding to that emotion will be played through the hologram.

[0291] Through the above specific processing steps, the system effectively utilizes data from the deceased person's lifetime and provides appropriate responses according to the user's emotional state, thereby achieving emotional healing.

[0292] Example 2

[0293] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0294] In today's world, there is a need for technology that can preserve the memory and presence of the deceased. However, conventional methods have difficulty reproducing the detailed characteristics of the deceased, and they are unable to respond to the user's emotional state. Therefore, there is a need for technology that allows users to feel deep healing through interactions with the deceased.

[0295] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing text data using natural language processing technology to extract characteristics of the deceased's words and actions; means for analyzing voice data using voice recognition technology to extract the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology to extract the deceased's appearance and movements; means for generating a hologram and voice model of the deceased using a generative AI based on the analyzed data; means for transferring the generated hologram and voice model to a hologram device; means for displaying a hologram of the deceased and allowing the user to interact with the deceased; means for analyzing the user's voice and facial expression data and using an emotion engine to identify the user's emotional state; and means for adjusting the hologram's behavior and responses based on the identified emotional state. This enables a high level of reproduction of the deceased's memories and enables appropriate responses tailored to the user's emotional state.

[0296] "Means for uploading data" means the ability for a user to access the system and send data to the system, such as audio, photographs, videos, social media posts, statements, writings, letters, etc., of the deceased.

[0297] "Means for verifying integrity and completeness" refers to the functionality by which the server verifies that data uploaded to the system is in the proper format and is not corrupted.

[0298] "Means of analyzing text data using natural language processing technology and extracting characteristics of speech and behavior" refers to technology for analyzing phrases and speaking styles frequently used by the deceased from uploaded text data.

[0299] "Means for analyzing audio data and extracting voice quality and speaking style" refers to technology that analyzes uploaded audio data and identifies the tone of the deceased's voice and speaking style characteristics.

[0300] "Means of analyzing images and videos to extract appearance and movements" refers to technology for detecting the facial features and physical movements of the deceased from uploaded photos and videos.

[0301] "Means for generating holograms and audio models using generative AI" refers to technology for generating 3D models and audio files of deceased people based on analyzed data.

[0302] The "means for transferring the hologram and audio model to the hologram device" is a function for packaging the generated data and sending it to the device that displays the hologram.

[0303] "Means for displaying and interacting with holograms" refers to a function that allows a user to use a hologram device to display and interact with a hologram of a deceased person.

[0304] "Means using an emotion engine" refers to technology that analyzes the user's voice and facial expressions to identify their current emotional state.

[0305] The "means for adjusting the hologram's actions and responses" is a function for appropriately changing the hologram's actions and speech based on the user's emotional state.

[0306] This invention is a system that uses data from the deceased's life to create a hologram using a generative AI model and combines it with an emotion engine to respond according to the user's emotional state. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[0307] Uploading data

[0308] Users access the system and upload data about the deceased. For example, a user accesses the system's website from a home computer or smartphone and uploads photos of their deceased grandfather or past messages. This uploaded data is sent to the server.

[0309] Receipt and confirmation of data

[0310] The server receives the uploaded data and checks its consistency and completeness. The server validates the uploaded file format (JPEG, PNG, etc.) and size (e.g., under 5MB), and ensures that images are loaded properly and audio files are not corrupted. This ensures that the data used by the system is available without any issues.

[0311] Data analysis

[0312] The server analyzes the received data and extracts the necessary information using the following techniques:

[0313] Natural language processing technology (NLP): Analyzes the deceased's frequently used phrases and speaking characteristics from uploaded text data.

[0314] Voice recognition technology: Analyzes voice data to determine the voice quality and speaking style of the deceased.

[0315] Image recognition technology: Analyzes the face and gestures of the deceased from photographs and videos.

[0316] For example, the server can understand the context of messages and letters from the deceased, identify common phrases and speaking characteristics, and even identify vocal characteristics from audio recordings.

[0317] Hologram generation

[0318] The AI ​​then generates a hologram of the deceased person based on the analyzed data, using the following techniques:

[0319] 3D modeling software (e.g. Blender): Create a 3D model of the deceased's appearance.

[0320] Speech synthesis software (e.g., Google® Text-to-Speech): Generates an audio file and recreates the voice of the deceased.

[0321] Motion capture technology (e.g., Kinect): Creating animations that replicate the movements of the deceased.

[0322] For example, generative AI can create a 3D model of your grandfather based on old photos, or generate natural-sounding conversational voices based on audio recordings.

[0323] Hologram data transfer

[0324] The server transfers the generated hologram data to the hologram device, packages the generated 3D model, audio file, and motion data, and sends the package to the hologram device. When the transfer is complete, the server records the completion of the data transfer.

[0325] User Notifications

[0326] The server notifies the user that the hologram data transfer has been completed. For example, the server sends a notification email to the user's terminal stating "Hologram data has been transferred to the device." It also displays a pop-up notification on the system's web page.

[0327] Hologram display

[0328] The user activates the hologram device and displays the hologram of the deceased. For example, the user turns on the hologram device, launches the application, and clicks the "Show hologram of the deceased" button, which displays the transferred hologram of the deceased.

[0329] Interaction and Emotion Recognition

[0330] Users can enjoy interacting with the hologram. By talking to the hologram or giving specific instructions, users can interact with the hologram and recreate memories of their deceased loved one. The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness."

[0331] Use of emotion engine

[0332] The hologram's behavior and responses are adjusted based on the identified emotional state. For example, if the user is sad, the hologram will provide a comforting message or a reminiscence. The emotion engine has the means to play pre-set memories and messages in response to specific emotional states. For example, if the user is moved, the hologram will deliver a warm message that corresponds to that emotion.

[0333] Prompt Sentence Examples

[0334] "Generate a hologram of the deceased person based on the following data: photographs, voice recordings, messages, letters, and design it to respond appropriately based on the user's emotional state."

[0335] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0336] Step 1:

[0337] The user accesses the system and uploads the deceased's data. The user accesses the system's website using a home computer or smartphone and clicks the "Upload" button. Next, on the upload screen, the user selects the deceased's audio, photos, videos, social media posts, words and actions, essays, letters, etc., and presses the "Send" button to send the data to the server. The input is the deceased's data file, and the output is the data uploaded to the server.

[0338] Step 2:

[0339] The server receives the uploaded data and checks its consistency and completeness. The server validates the received file format (JPEG, PNG, MP3, etc.) and size (e.g., less than 5MB), and ensures that images are loaded in the correct format and audio files are not corrupted. For example, it checks the header of a JPEG file to ensure that the image is valid. The input is the uploaded data file, and the output is the validated data file.

[0340] Step 3:

[0341] The server analyzes the received data and extracts the necessary information. Specifically, it uses natural language processing technology to analyze the text data and extract characteristics of the deceased's words and actions. It also uses speech recognition technology to analyze the audio data and extract the deceased's voice quality and speaking style, and image recognition technology to extract the deceased's appearance and movements from photos and videos. For example, it extracts frequently used phrases from the deceased's text messages. The input is verified data, and the output is analyzed data (text features, audio features, and image features).

[0342] Step 4:

[0343] The generative AI generates a hologram and voice model of the deceased based on the analyzed data. It creates a 3D model of the deceased using 3D modeling software (e.g., Blender), generates an audio file using speech synthesis software (e.g., Google Text-to-Speech), and creates animation data using motion capture technology (e.g., Kinect). The input is the analyzed data, and the output is the generated 3D model and audio file.

[0344] Step 5:

[0345] The server transfers the generated hologram data to the hologram device. The server then combines the generated 3D model, audio file, and motion data into a single package and transfers the package to the hologram device. For example, the server sends the data using the FTP protocol. The input is the generated hologram data, and the output is the data transferred to the hologram device.

[0346] Step 6:

[0347] The server notifies the user that the hologram data transfer is complete. The server sends a notification email to the user's email address stating "Hologram data has been transferred to the device" and also displays a pop-up notification on the system's web page. The input is the transfer completion status, and the output is the notification email and pop-up notification to the user.

[0348] Step 7:

[0349] The user turns on the hologram device and displays a hologram of the deceased. The user turns on the hologram device, launches the application, and clicks the "Show hologram of the deceased" button. The input is the hologram device, and the output is the displayed hologram of the deceased.

[0350] Step 8:

[0351] Users can enjoy interacting with holograms. By speaking to the hologram or giving specific instructions, users can interact with the hologram and recreate memories of their deceased loved ones. The input is the user's voice and actions, and the output is the hologram's response.

[0352] Step 9:

[0353] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness." The input is the user's voice and facial expression data, and the output is the identified emotional state.

[0354] Step 10:

[0355] The hologram's behavior and response are adjusted based on the identified emotional state. For example, if the user is sad, the hologram may offer comforting messages such as, "Are you OK? Is something wrong?" The input is the identified emotional state, and the output is the adjusted hologram's behavior and response.

[0356] (Application example 2)

[0357] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0358] In a hologram system based on data from the deceased's lifetime, when a user enjoys interacting with a hologram of the deceased, it is necessary to obtain appropriate responses according to the user's emotional state. In particular, when displaying holograms using smart glasses in a physical store, it is necessary to realize natural responses and interactions according to changes in the user's emotions.

[0359] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing text data using natural language processing technology and extracting characteristics of the deceased's words and actions; means for analyzing voice data using voice recognition technology and extracting the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology and extracting the deceased's appearance and movements; means for generating a hologram and voice model based on the deceased's data before death and the user's emotional state; means for transferring the generated hologram and voice model to a hologram device; means for the user to enjoy interaction with the deceased in a physical store using smart glasses; means for analyzing the user's voice and face data using an emotion engine and identifying the user's emotional state; and means for adjusting the hologram's behavior and response based on the identified emotional state. This makes it possible to achieve optimal responses and natural interactions according to the user's emotional state.

[0360] "Voice of the deceased" refers to voice data recorded by the deceased during their lifetime.

[0361] A "photograph" is a still image of the deceased or related objects or people.

[0362] "Video" is dynamic image data of the deceased or related objects or people.

[0363] "Social media posts" are content such as text, images, and videos posted by the deceased on social media while they were alive.

[0364] "Words and actions" is a record of the words and actions of the deceased during their lifetime.

[0365] "Compositions" refer to essays and writings written by the deceased during their lifetime.

[0366] A "letter" is a letter or message written by a deceased person while they were still alive.

[0367] "Means for uploading" refers to the procedures and interfaces that allow users to send data about the deceased to the server.

[0368] "Integrity and completeness check measures" are processes that verify that uploaded data is in the correct format and is complete.

[0369] "Natural language processing technology" is a technology for analyzing text data and extracting meaning and characteristics.

[0370] "Voice recognition technology" is a technology that analyzes voice data to identify the speaker's voice quality and speaking style.

[0371] "Image recognition technology" is a technology that analyzes images and videos to extract the appearance and movements of a target.

[0372] An "emotion engine" is a technology that identifies a user's emotional state from their voice and facial expressions and adjusts responses accordingly.

[0373] A "hologram device" is a device for displaying a generated hologram.

[0374] "Smart glasses" are augmented reality (AR) display devices worn by users.

[0375] "Means for enjoying interaction" are functions and processes that allow users to interact with holograms through dialogue and manipulation.

[0376] The system for carrying out the present invention includes the following processing flow and data analysis process.

[0377] 1. Upload your data

[0378] Users access the system using their home computers or smartphones and upload data such as audio, photos, videos, social media posts, words and actions, essays, and letters of the deceased, thereby collecting information about the deceased's activities and characteristics while they were alive.

[0379] 2. Receipt and confirmation of data

[0380] The server receives the data uploaded by the user and checks the consistency and completeness of each data item. This includes validating the file format and size, and testing the readability of image and audio files. For example, the server checks whether the received image file is corrupted.

[0381] 3. Data Analysis

[0382] The server analyzes the received data and extracts the required information using the following techniques:

[0383] Natural language processing technology (NLP): Analyzes the characteristics of the deceased's words and actions from text data.

[0384] Voice recognition technology: Analyzes the voice quality and speaking style of the deceased from audio data.

[0385] Image recognition technology: Analyzes the face and gestures of the deceased from photographs and videos.

[0386] 4. Hologram Generation

[0387] Based on the deceased's life data and the user's emotional state, the generative AI generates a hologram and voice model. Specifically, it uses 3D modeling technology to recreate the deceased's appearance, voice synthesis technology to generate the deceased's voice, and motion capture technology to recreate their movements and behavior.

[0388] 5. Transfer of hologram data

[0389] The server transfers the generated hologram data to the hologram device, which includes a package of 3D models, audio files, and motion data. After the transfer is complete, the server notifies the user of the transfer to their smart glasses.

[0390] 6. Interaction and Emotion Recognition

[0391] Users interact with holograms in brick-and-mortar stores using smart glasses. The smart glasses' cameras and microphones capture the user's voice and facial expressions and transmit them to a server. The server uses an emotion engine to analyze the user's emotional state and adjust the hologram's responses and behavior based on the identified emotion. For example, if the user has a sad expression, the hologram will convey a comforting message such as "Cheer up."

[0392] Examples of prompt statements

[0393] Prompt statement:

[0394] Recognize the user's emotional state and generate a hologram that responds based on that state. Take the following data as input:

[0395] 1. User's voice data: voice_sample.wav

[0396] 2. User's facial expression image: face_sample.png

[0397] 3. Text data of the deceased:grandfather_messages.txt

[0398] The generated hologram will have animation and audio that will convey an uplifting message if the user is feeling sad.

[0399] This allows users to recreate memories of the deceased in a physical store and receive appropriate responses based on their emotions.

[0400] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0401] Step 1:

[0402] Users access the system using their home computers or smartphones and upload data about the deceased, such as audio, photos, videos, social media posts, words and actions, essays, letters, etc. The input is various data related to the deceased, and the output is this data stored on the server.

[0403] Step 2:

[0404] The server receives the uploaded data and checks the consistency and completeness of each piece of data. The input is the file uploaded by the user, and the output is the verified data. Specifically, it validates the file format and size, and performs read tests on image and audio files. For example, the server checks whether the received image file is corrupted.

[0405] Step 3:

[0406] The server uses natural language processing technology to analyze text data and extract characteristics of the deceased's words and actions. The input is text data, and the output is data that represents the characteristics of the deceased's words and actions. Specifically, it uses an NLP library to perform context analysis and key phrase extraction.

[0407] Step 4:

[0408] The server uses voice recognition technology to analyze the voice data and extract the voice quality and speaking style of the deceased. The input is the voice data, and the output is characteristic data of the voice quality and speaking style of the deceased. Specifically, it uses a voice recognition library to analyze pitch and tone from the voice waveform.

[0409] Step 5:

[0410] The server uses image recognition technology to analyze photos and videos to extract the appearance and movements of the deceased. The input is photo and video data, and the output is data for recreating the appearance and movements of the deceased. Specific operations include facial recognition and motion capture using an image processing library.

[0411] Step 6:

[0412] The server uses generative AI to generate a hologram and voice model based on the deceased's data and the user's emotional state. The input is the analyzed characteristic data of the deceased and the user's emotional data, and the output is the hologram and voice model data. Specifically, the hologram is created using 3D modeling, voice synthesis, and motion capture technology.

[0413] Step 7:

[0414] The server packages the generated hologram data and transfers it to the hologram device. The input is the generated hologram and audio model data, and the output is data in a format that can be displayed on the hologram device. Specifically, data compression and transfer protocols are used to ensure rapid data transfer.

[0415] Step 8:

[0416] The user enjoys interacting with the deceased person in a physical store using smart glasses. The input is data transmitted to the hologram device, and the output is a displayed hologram. Specifically, the user launches the smart glasses application and begins interacting with the displayed hologram.

[0417] Step 9:

[0418] The user's voice and facial expression data are captured by the smart glasses and sent to the server. The input is the user's voice and facial expression data, and the output is trigger data to start the analysis. Specifically, the data is captured using the microphone and camera of the smart glasses and sent to the server via the network.

[0419] Step 10:

[0420] The server uses an emotion engine to analyze the user's voice and facial expression data to identify the user's emotional state. The input is the user's voice and facial expression data, and the output is the user's emotional state data. Specifically, the server applies voice analysis and facial expression analysis algorithms to analyze emotions.

[0421] Step 11:

[0422] The server adjusts the hologram's behavior and response based on the identified emotional state. The input is the user's emotional state data, and the output is the adjusted hologram response data. Specifically, it uses an AI model to generate and display emotionally appropriate hologram responses.

[0423] Examples of prompts:

[0424] Prompt statement:

[0425] Recognize the user's emotional state and generate a hologram that responds based on that state. Take the following data as input:

[0426] 1. User's voice data: voice_sample.wav

[0427] 2. User's facial expression image: face_sample.png

[0428] 3. Text data of the deceased:grandfather_messages.txt

[0429] The generated hologram will have animation and audio that will convey an uplifting message if the user is feeling sad.

[0430] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0431] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0432] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0433] [Second embodiment]

[0434] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0435] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0436] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0437] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0438] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0440] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0441] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0442] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0443] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0444] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0445] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0446] This invention is a system that uses generative AI to create holograms based on data from the deceased's life, providing comfort to surviving family members and loved ones. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[0447] Uploading data

[0448] The user accesses the system and uploads the necessary data on the deceased.

[0449] Example: A user accesses the system on their home computer or smartphone and uploads photos of their deceased grandfather and past messages.

[0450] Receipt and confirmation of data

[0451] The server receives the uploaded data and verifies its consistency and completeness.

[0452] For example: A server might validate uploaded file types and sizes, ensure images load properly, and ensure audio files aren't corrupted.

[0453] Data analysis

[0454] The server analyzes the received data and extracts the necessary information.

[0455] Using natural language processing (NLP) technology, the characteristics of the deceased's words and actions are analyzed from text data.

[0456] Using voice recognition technology, characteristic voice qualities and speaking styles are analyzed from audio data.

[0457] Image recognition technology is used to analyze the face and gestures of the deceased from photographs and videos.

[0458] Example: The server understands the context of messages and letters from the deceased and identifies common phrases and speaking characteristics.

[0459] Hologram generation

[0460] The generating AI generates a hologram of the deceased person based on the analyzed data.

[0461] Using 3D modeling technology, a 3D model is created that recreates the appearance of the deceased.

[0462] Using voice synthesis technology, an audio file is created that reproduces the voice of the deceased.

[0463] Motion capture technology is used to create animations that recreate the movements of the deceased.

[0464] Example: Generative AI creates a 3D model of your grandfather based on old photos and generates natural-sounding conversational voices based on audio recordings.

[0465] Hologram data transfer

[0466] The server transfers the generated hologram data to the hologram device.

[0467] The generated 3D model, audio file, and motion data are packaged into a single file and transferred to the hologram device.

[0468] Example: The server notifies the user that "the hologram of your grandfather is complete" and transfers the data to the device.

[0469] Hologram display

[0470] The user activates the hologram device, which displays a hologram of the deceased person.

[0471] Turn on the hologram device and start the specified application.

[0472] Example: A user turns on a hologram device in their living room and relives a holographic story their grandfather used to tell their family.

[0473] Interaction and Continued Use

[0474] Users enjoy interacting with the holograms.

[0475] Users can talk to the hologram and carry on a conversation.

[0476] New data analyses and topics are added to the hologram periodically.

[0477] Example: A user can ask a hologram, "What story would you like to tell us today?" and the hologram will relive a story from an old family photo.

[0478] As a result, this system can effectively utilize data from the deceased's lifetime, providing deep healing for family members and those involved.

[0479] The processing flow will be explained below.

[0480] Step 1:

[0481] Users access the system and upload data about the deceased. Specifically, they log in and open the data upload page. From there, they select files such as photos, audio, video, social media posts, essays, and letters, and press the upload button to send them to the server.

[0482] Step 2:

[0483] The server receives the uploaded data. Specifically, the server receives the data file sent by the user and saves it in a specified storage location.

[0484] Step 3:

[0485] The server checks the integrity and completeness of the data it receives. Specifically, it checks the file format, size, and data corruption status to check for any abnormalities. If an abnormality is found, it sends a notification to the user urging them to re-upload.

[0486] Step 4:

[0487] The server analyzes the data. Specifically, it uses natural language processing (NLP) technology to extract characteristics of the deceased's words and actions from the text data. It also uses a language model to analyze frequently occurring phrases and context.

[0488] Step 5:

[0489] The server uses voice recognition technology to analyze the audio data, extracting characteristics of the deceased's voice quality and speaking style from the audio data and storing these characteristics in a database.

[0490] Step 6:

[0491] The server uses image recognition technology to analyze photos and videos, extracting facial features and gestures of the deceased, and gathers the information needed to generate a 3D model.

[0492] Step 7:

[0493] Based on the analyzed data, the AI ​​generates a hologram of the deceased. Specifically, it uses 3D modeling technology to create a 3D model that recreates the appearance of the deceased, voice synthesis technology to generate an audio file that recreates the deceased's voice, and motion capture technology to create an animation that recreates the movements of the deceased.

[0494] Step 8:

[0495] The server transfers the generated hologram data to the hologram device. Specifically, it packages the generated 3D model, audio file, and motion data, and transfers the package to the hologram device.

[0496] Step 9:

[0497] The server notifies the user that the hologram data transfer has been completed. Specifically, the server notifies the user by sending a pop-up screen or email to the user's device stating that "the hologram data has been transferred to the device."

[0498] Step 10:

[0499] The user activates the hologram device and displays the hologram. Specifically, the user turns on the device, launches the application, and displays the transferred hologram of the deceased person.

[0500] Step 11:

[0501] Users can enjoy interacting with the hologram by speaking to it or giving it specific instructions, thereby recreating memories of the deceased.

[0502] Through the above processing steps, a hologram based on the data of the deceased person is generated, and the user can find emotional healing through it.

[0503] Example 1

[0504] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0505] In modern society, people deeply cherish memories of their deceased loved ones, but there are limited ways to physically experience them. Traditional memorial methods struggle to recreate the voices and movements of the deceased, resulting in the bereaved family members being unable to feel close to them. In particular, there is a need for a method to heal the hearts of bereaved family members by recreating the voices and movements of the deceased.

[0506] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0507] In this invention, the server includes: means for a user to upload data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for the server to verify the consistency and completeness of the uploaded data; means for the server to analyze the text data using natural language processing technology and extract characteristics of the deceased's words and actions; means for the server to analyze the voice data using voice recognition technology and extract the deceased's voice quality and speaking style; means for the server to analyze photos and videos using image recognition technology and extract the deceased's appearance and movements; means for the generation AI to generate a hologram and a voice model based on the generated characteristics of the deceased; means for the server to transfer the hologram and voice model to a hologram device; and means for the user to display a hologram of the deceased and interact with the deceased. This allows the user to interact with the hologram of the deceased and experience a physical sensation of the deceased.

[0508] "User" refers to an individual who accesses the system, uploads data of a deceased person, and uses a hologram device.

[0509] "Server" refers to the device or system that receives, verifies, and analyzes data uploaded by users, and generates and transmits holograms and audio models.

[0510] A "deceased person" refers to a person who has already passed away, but about whom the user wishes to retain memories.

[0511] "Audio data" refers to files containing recordings of the deceased person's voice.

[0512] "Photograph" refers to a still image that visually records the appearance of the deceased.

[0513] "Video" refers to a dynamic image file that includes the movements of the deceased.

[0514] "Social media posts" refers to data such as text, images, and videos posted by the deceased on social media platforms on the Internet.

[0515] "Words and actions" refers to the words and actions of the deceased.

[0516] "Composition" refers to a document written by the deceased.

[0517] A "letter" refers to a document written by a deceased person as a form of personal communication.

[0518] "Data Integrity" refers to the state in which uploaded data is accurate and complete.

[0519] "Natural language processing technology" refers to the technology of analyzing text data and extracting meaning from it.

[0520] "Voice recognition technology" refers to technology that analyzes voice data and extracts features from the voice.

[0521] "Image recognition technology" refers to the technology of analyzing images and videos and extracting features from them.

[0522] "Generative AI" refers to artificial intelligence technology that automatically generates holograms and audio models based on analyzed data.

[0523] "Hologram device" refers to a device that displays generated holograms and allows a user to interact with them.

[0524] "Interaction" refers to a two-way exchange between a user and a hologram.

[0525] The present invention is a system that uses AI to create holograms based on data from the deceased person's life, providing comfort to surviving family members and those involved. Specific embodiments for carrying out the present invention are described below.

[0526] Uploading data

[0527] The user accesses the system and uploads the necessary data about the deceased. This is accomplished by logging into the system's web portal using a home computer or smartphone and uploading photos, audio files, videos, text files, etc. of the deceased. For example, a user can select photos of their deceased grandfather and past messages to be imported into the system.

[0528] Receipt and confirmation of data

[0529] The server receives the uploaded data and checks its consistency and integrity. Specifically, it checks the file format (e.g., jpg, mp3, mp4, txt, etc.) of each file received by the server, and then calculates the file's hash value to ensure the data is not corrupted. If an improper file format or corruption is detected, an error message is returned to the user.

[0530] Data analysis

[0531] The server analyzes the received data and extracts the necessary information. Natural language processing (NLP) technology is used to analyze the text data and extract characteristics of the deceased's words and actions. For example, distinctive phrases and speaking characteristics are extracted from letters and emails uploaded by the user. Speech recognition technology is also used to analyze the audio data and identify the deceased's voice quality and intonation. Image recognition technology is used to analyze the deceased's face and movements from photos and videos. The specific software used at this stage is Python libraries (NLTK, SpaCy, OpenCV, etc.) and machine learning frameworks such as TensorFlow.

[0532] Hologram generation

[0533] The generative AI generates a hologram of the deceased based on the analyzed data. 3D modeling technology is used to create a 3D model that replicates the appearance of the deceased. Voice synthesis technology is used to create an audio file that replicates the deceased's voice. Motion capture technology is then used to create an animation that replicates the natural movements of the deceased. Specific software used at this stage includes Blender (3D modeling) and Vosk (voice synthesis). The prompt is something like, "Generate a 3D hologram of my grandfather based on his old photos and a recorded message."

[0534] Hologram data transfer

[0535] The server transfers the generated hologram data to the hologram device. The generated 3D model, audio file, and motion data are packaged and transferred to the hologram device as a single file. Once the transfer is complete, the user is notified.

[0536] Hologram display and interaction

[0537] The user activates the hologram device, which displays a hologram of the deceased. The user turns on the device and launches a specified application. The hologram device reads the data transferred from the server and displays a 3D hologram of the deceased. It plays audio and runs animations in real time. The user can talk to the hologram, which responds by reflecting the characteristics of the deceased. Enjoying this interaction brings back fond memories of the deceased and brings emotional healing.

[0538] In this way, the invention utilizes data from the deceased person's lifetime to provide a hologram that is extremely valuable to their family and friends.

[0539] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0540] Step 1:

[0541] Users access the system and upload data such as audio, photos, videos, social media posts, words, actions, essays, and letters of the deceased. The input is various data files (e.g., jpg, mp3, mp4, txt, etc.) owned by the user, and the output is the data files imported into the system. Specifically, users log in to the system's web portal using their home computer or smartphone and perform the upload operation.

[0542] Step 2:

[0543] The server receives data uploaded by users and checks its integrity and completeness. The input is the data file uploaded by the user, and the output is the data file whose integrity and completeness have been checked. Specifically, the server verifies the format of each file (e.g., jpg, mp3, mp4, etc.) and calculates a hash value to ensure the file is not corrupted. If the file is corrupted or in an improper format, an error message is returned to the user.

[0544] Step 3:

[0545] The server analyzes the received data and extracts the necessary information. The input is a data file whose consistency and completeness has been confirmed, and the output is a dataset with extracted characteristics of the deceased. Specific operations include using NLP technology to analyze the text data and extract commonly used phrases and speaking characteristics; analyzing the audio data using speech recognition technology to identify the voice quality and intonation of the deceased; and using image recognition technology to extract the face and movements of the deceased from photos and videos. Specific software used is Python libraries (NLTK, SpaCy, OpenCV, etc.) and TensorFlow.

[0546] Step 4:

[0547] The generation AI generates a hologram of the deceased based on the analyzed data. The input is the characteristic data of the deceased analyzed on the server, and the output is a hologram file containing a 3D model, audio file, and motion data. Specific operations involve using 3D modeling technology to create a 3D model that recreates the appearance of the deceased, and using voice synthesis technology to generate an audio file that recreates the deceased's voice. Furthermore, motion capture technology is used to create an animation that recreates the movements of the deceased. The specific software used is Blender (3D modeling) and Vosk (voice synthesis).

[0548] Step 5:

[0549] The server transfers the generated hologram data to the hologram device. The input is the generated hologram file, and the output is the data stored in the hologram device. Specifically, the server packages the generated 3D model, audio file, and motion data and transfers it to the hologram device as a single file. Once the transfer is complete, the server notifies the user that the data transfer is complete.

[0550] Step 6:

[0551] The user starts the hologram device and displays a hologram of the deceased. The input is a hologram file stored in the hologram device, and the output is a hologram of the deceased that is displayed and played back. Specifically, the user turns on the hologram device and launches a specified application. The hologram device reads the data, displays a 3D hologram of the deceased, plays audio in real time, and runs animations.

[0552] Step 7:

[0553] Users can enjoy interacting with the hologram. The input is responses from the hologram device and user input (voice and gestures), and the output is an interactive exchange experience. Specifically, when the user speaks to the hologram, the hologram responds using pre-analyzed data and real-time voice recognition technology. Continuing conversation and interaction with the user can revive fond memories of the deceased, bringing emotional healing.

[0554] (Application example 1)

[0555] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0556] Systems that more realistically recreate memories of the deceased and provide comfort to surviving family and friends face many technical challenges. In particular, systems that not only accurately reflect the characteristics of the deceased but also deepen communication with users through real interactions are required. Another challenge is to personalize the shopping experience and provide special value by recommending products based on the user's purchasing preferences.

[0557] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0558] In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing the text data using natural language processing technology to extract characteristics of the deceased's words and actions; means for analyzing the voice data using voice recognition technology to extract the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology to extract the deceased's appearance and movements; means for generating a hologram and a voice model based on the generated characteristics of the deceased; means for transferring the hologram and the voice model to a hologram device; means for displaying a hologram of the deceased and allowing the user to interact with the deceased; and means for recommending products based on the user's purchasing preferences. This enables the user's purchasing experience to be personalized while realistically recreating the deceased's memories.

[0559] A "deceased person" refers to a person who has passed away and is the person about whom data is to be reproduced.

[0560] "Data upload" refers to a user sending information about a deceased person (such as audio, photos, videos, social media posts, words, actions, writings, letters, etc.) to the system.

[0561] "Data integrity and completeness" means ensuring that uploaded data is accurate, complete, and not corrupted or missing.

[0562] "Natural language processing technology" is a technology that analyzes text data and extracts the characteristics and meaning of sentences and behavior.

[0563] "Voice recognition technology" is a technology that analyzes voice data and extracts characteristics such as speaking style and voice quality.

[0564] "Image recognition technology" is a technology that analyzes photographic and video data to extract a person's appearance and movements.

[0565] A "hologram" is a three-dimensional image reproduced to recreate the appearance and movements of a deceased person.

[0566] A "voice model" is digital voice data generated to reproduce the voice of a deceased person.

[0567] A "hologram device" is a hardware device for displaying generated holograms and audio models.

[0568] "Interaction" means that the user can have a conversation or dynamic interaction with the hologram of the deceased person.

[0569] "Purchasing preferences" refers to purchasing tendencies based on a user's interests and preferences.

[0570] "Product recommendation" refers to the process of presenting relevant products based on a user's purchasing preferences.

[0571] The system for implementing this invention generates a hologram and a voice model using various data of the deceased person, and allows the user to interact with the hologram and voice model and receive personalized product recommendations. Specific embodiments are described below.

[0572] System Overview

[0573] The system consists of the following main components:

[0574] 1. Data upload method

[0575] 2. Means of verifying data consistency and completeness

[0576] 3. Data analysis methods (natural language processing technology, voice recognition technology, image recognition technology)

[0577] 4. Hologram and audio model generation method

[0578] 5. Transfer method to hologram device

[0579] 6. Holographic Display and Interaction Methods

[0580] 7. Product recommendation means

[0581] Hardware and software used

[0582] Hardware:

[0583] Common devices such as smartphones, tablets, and personal computers

[0584] Dedicated hologram device

[0585] software:

[0586] Server Application

[0587] Natural Language Processing (NLP) Engine

[0588] Speech Recognition Engine

[0589] Image Recognition Engine

[0590] Generative AI Models

[0591] Speech synthesis engine

[0592] Product recommendation engine

[0593] Operation flow

[0594] 1. Data upload:

[0595] Users use their own devices (such as smartphones) to upload data such as audio, photos, videos, social media posts, essays, and letters of the deceased to the system.

[0596] 2. Data consistency and integrity checks:

[0597] The server checks the integrity and completeness of the uploaded data, specifically by validating the file format, size, and content, and prompting the user to re-upload if there are any issues.

[0598] 3. Data Analysis:

[0599] The server will analyze the uploaded data using the following analysis techniques:

[0600] Text analysis: Using NLP techniques to extract characteristics of the deceased's words and actions from text data.

[0601] Speech analysis: Using speech recognition technology to extract voice quality and speaking style from audio data.

[0602] Image analysis: Using image recognition technology to analyze the appearance and movements of the deceased from photographs and videos.

[0603] 4. Hologram and audio model generation:

[0604] A generative AI model is used to generate a hologram and voice model of the deceased person based on the acquired feature data.

[0605] 5. Transfer to hologram device:

[0606] The generated hologram and audio model are packaged and transferred to the hologram device.

[0607] 6. Hologram Display and Interaction:

[0608] The user activates the hologram device and interacts with the hologram of the deceased person. The hologram responds to the user's input, realizing a dialogue with the user.

[0609] 7. Product recommendation:

[0610] Using a product recommendation engine, the hologram suggests products based on the user's purchasing preferences, just like a shop assistant, and explains the reasons and features of the recommended products.

[0611] Prompt Sentence Examples

[0612] Examples:

[0613] User: The user opens a shopping app on their smartphone and enjoys shopping while viewing a hologram of their deceased mother. When the user says, "I want a new shirt," the hologram suggests several shirts and explains why it recommends them.

[0614] Example prompt sentence:

[0615] text

[0616] Show me a hologram of a deceased person who knows my preferences and suggests products in a specific category (e.g., "shirts"), how I chose them, and why I recommend them.

[0617] Using this system, users can not only recreate memories of their deceased loved ones, but also have a personalized shopping experience through interactions with the deceased.

[0618] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0619] Step 1:

[0620] A user uses a device (smartphone or tablet) to upload data about the deceased (audio, photos, videos, social media posts, words, actions, essays, letters, etc.) to the system. These files are sent to the server as input data. The server temporarily stores the received data as output.

[0621] Step 2:

[0622] The server verifies the integrity and completeness of the uploaded data. As input, it validates the uploaded data file, checking the file format, size, whether it is corrupted, and ensuring that all data has been received correctly. As output, it generates a flag indicating whether the data is valid.

[0623] Step 3:

[0624] The server analyzes the received data. The input data is the deceased person's data received in step 1. Specifically, the following process is performed:

[0625] Using natural language processing technology, characteristics of the deceased's words and actions are extracted from text data.

[0626] Using voice recognition technology, voice quality and speaking style are extracted from audio data.

[0627] Using image recognition technology, the appearance and movements of the deceased are extracted from photographs and videos.

[0628] As an output, feature data of the deceased extracted by each technique is generated.

[0629] Step 4:

[0630] A generative AI model is used to generate a hologram and voice model based on the analyzed feature data. The feature data obtained in step 3 is used as input. Specifically, the appearance of the deceased is recreated using 3D modeling software, and the voice is recreated using voice synthesis technology. The output is hologram data and voice data.

[0631] Step 5:

[0632] The server transfers the generated hologram and audio data to the hologram device. As input, the data generated in step 4 is used. As output, it is sent to the hologram device, where it is converted into a displayable format.

[0633] Step 6:

[0634] The user activates the hologram device and displays a hologram of the deceased person. As input, hologram and audio data are loaded into the hologram device. Specifically, the user turns on the device and launches a dedicated application. As output, a hologram of the deceased person is displayed and interaction begins.

[0635] Step 7:

[0636] The hologram recommends products based on the user's purchasing preferences. The user's purchasing history and preference data are used as input. Specifically, the product recommendation engine analyzes the user's preferences and generates a product list based on them. As output, the hologram explains to the user the recommended products, their features, and the reasons for the recommendation.

[0637] Prompt Sentence Examples

[0638] Show me a hologram of a deceased person who knows my preferences and suggests products in a specific category (e.g., "shirts"), how I chose them, and why I recommend them.

[0639] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0640] This invention is a system that uses generative AI to create a hologram based on data from the deceased's life, and combines it with an emotion engine to respond according to the user's emotional state. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[0641] Uploading data

[0642] A user accesses the system and uploads data about the deceased. For example, a user accesses the system on a home computer or smartphone and uploads photos of a deceased grandfather or past messages.

[0643] Receipt and confirmation of data

[0644] The server receives the uploaded data and checks its consistency and completeness. For example, the server validates the uploaded file format and size, ensures that images load properly, and that audio files are not corrupted.

[0645] Data analysis

[0646] The server analyzes the received data and extracts the necessary information.

[0647] Using natural language processing (NLP) technology, the characteristics of the deceased's words and actions are analyzed from text data.

[0648] Using voice recognition technology, characteristic voice qualities and speaking styles are analyzed from audio data.

[0649] Using image recognition technology, the server analyzes the face and body language of the deceased from photos and videos. For example, the server can understand the context of messages and letters from the deceased and identify common phrases and speaking characteristics.

[0650] Hologram generation

[0651] The generating AI generates a hologram of the deceased person based on the analyzed data.

[0652] 3D modeling technology is used to create a 3D model that recreates the appearance of the deceased.

[0653] Voice synthesis technology is used to create an audio file that reproduces the voice of the deceased.

[0654] Motion capture technology is used to create animations that replicate the movements of the deceased, while generative AI creates a 3D model of your grandfather based on old photos, and generates natural-sounding conversational voices based on audio recordings.

[0655] Hologram data transfer

[0656] The server transfers the generated hologram data to the hologram device, packages the generated 3D model, audio file, and motion data, and transfers the package to the hologram device.

[0657] User Notifications

[0658] The server notifies the user that the hologram data transfer has been completed. For example, the server notifies the user by sending a message to the user's device via a pop-up screen or email saying "The hologram data has been transferred to your device."

[0659] Hologram display

[0660] The user activates the hologram device and displays the hologram of the deceased person, e.g., by turning on the device, launching an application, and displaying the transferred hologram of the deceased person.

[0661] Interaction and Emotion Recognition

[0662] Users can enjoy interacting with the holograms. By speaking to the holograms or giving them specific instructions, users can interact with the holograms and relive memories of their deceased loved ones.

[0663] Use of emotion engine

[0664] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness."

[0665] The hologram's behavior and responses are tailored based on the identified emotional state: for example, if the user is sad, the hologram will offer a comforting message or a reminiscence.

[0666] The emotion engine has a means to play pre-set memories and messages according to a specific emotional state. For example, if the user is emotional, the hologram will convey a warm message according to that emotion.

[0667] As a result, this system effectively utilizes data from the deceased person's lifetime and provides optimal responses according to the user's emotional state, thereby achieving deep healing.

[0668] The processing flow will be explained below.

[0669] Step 1:

[0670] Users access the system and upload data about the deceased. For example, they log in to a dedicated web application on their home computer or smartphone, select files such as photos, audio, video, social media posts, essays, and letters of the deceased, and press the upload button to send them to the server.

[0671] Step 2:

[0672] The server receives the uploaded data and checks its integrity and completeness. Specifically, it checks the file format and size of the data received, ensures that images are loaded properly, and that audio files are not corrupted. If there are any abnormalities, it sends a notification to the user, encouraging them to re-upload.

[0673] Step 3:

[0674] The server analyzes the received data. Specifically,

[0675] Using natural language processing (NLP) technology, characteristics of the deceased's words and actions are extracted from text data (letters, social media posts, etc.).

[0676] Voice recognition technology is used to extract the voice quality and speaking style of the deceased from audio data (such as recordings).

[0677] Image recognition technology is used to extract facial and movement characteristics of the deceased from photographs and videos.

[0678] Step 4:

[0679] The AI ​​generates a hologram of the deceased person based on the analyzed data.

[0680] Using 3D modeling technology, a 3D model is created that recreates the appearance of the deceased based on collected facial and body features.

[0681] Using voice synthesis technology, an audio file is created that reproduces the voice of the deceased based on the extracted voice quality and speaking characteristics.

[0682] Motion capture technology is used to create animations that recreate the movements of the deceased.

[0683] Step 5:

[0684] The server transfers the generated hologram data to the hologram device. Specifically, it combines the generated 3D model, audio file, and motion data into a single package and sends it to the hologram device.

[0685] Step 6:

[0686] The server notifies the user that the hologram data transfer has been completed. Specifically, the server sends a pop-up notification or email to the user's device stating, "Hologram data has been transferred to your device."

[0687] Step 7:

[0688] The user activates the hologram device and displays the hologram of the deceased. Specifically, the user turns on the hologram device, launches the dedicated application, reads the transferred hologram data, and displays the hologram of the deceased.

[0689] Step 8:

[0690] Users can enjoy interacting with holograms. By talking to the hologram or giving specific instructions, the hologram will respond, establishing a dialogue.

[0691] Step 9:

[0692] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. Specifically, the hologram device captures the user's facial expressions and tone of voice through a built-in camera and microphone, and the emotion engine analyzes the data to determine emotions such as "sadness," "joy," and "surprise."

[0693] Step 10:

[0694] The server adjusts the hologram's behavior and responses based on the identified emotional state. For example, if the user is sad, the hologram will offer comforting words or warm messages, including reminiscences. If the user is happy, the hologram will engage in proactive interactions tailored to that emotion.

[0695] Step 11:

[0696] The server plays pre-set memories and messages according to the emotional state identified by the emotion engine. For example, if the user is emotional, a special message corresponding to that emotion will be played through the hologram.

[0697] Through the above specific processing steps, the system effectively utilizes data from the deceased person's lifetime and provides appropriate responses according to the user's emotional state, thereby achieving emotional healing.

[0698] Example 2

[0699] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0700] In today's world, there is a need for technology that can preserve the memory and presence of the deceased. However, conventional methods have difficulty reproducing the detailed characteristics of the deceased, and they are unable to respond to the user's emotional state. Therefore, there is a need for technology that allows users to feel deep healing through interactions with the deceased.

[0701] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing text data using natural language processing technology to extract characteristics of the deceased's words and actions; means for analyzing voice data using voice recognition technology to extract the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology to extract the deceased's appearance and movements; means for generating a hologram and voice model of the deceased using a generative AI based on the analyzed data; means for transferring the generated hologram and voice model to a hologram device; means for displaying a hologram of the deceased and allowing the user to interact with the deceased; means for analyzing the user's voice and facial expression data and using an emotion engine to identify the user's emotional state; and means for adjusting the hologram's behavior and responses based on the identified emotional state. This enables a high level of reproduction of the deceased's memories and enables appropriate responses tailored to the user's emotional state.

[0702] "Means for uploading data" means the ability for a user to access the system and send data to the system, such as audio, photographs, videos, social media posts, statements, writings, letters, etc., of the deceased.

[0703] "Means for verifying integrity and completeness" refers to the functionality by which the server verifies that data uploaded to the system is in the proper format and is not corrupted.

[0704] "Means of analyzing text data using natural language processing technology and extracting characteristics of speech and behavior" refers to technology for analyzing phrases and speaking styles frequently used by the deceased from uploaded text data.

[0705] "Means for analyzing audio data and extracting voice quality and speaking style" refers to technology that analyzes uploaded audio data and identifies the tone of the deceased's voice and speaking style characteristics.

[0706] "Means of analyzing images and videos to extract appearance and movements" refers to technology for detecting the facial features and physical movements of the deceased from uploaded photos and videos.

[0707] "Means for generating holograms and audio models using generative AI" refers to technology for generating 3D models and audio files of deceased people based on analyzed data.

[0708] The "means for transferring the hologram and audio model to the hologram device" is a function for packaging the generated data and sending it to the device that displays the hologram.

[0709] "Means for displaying and interacting with holograms" refers to a function that allows a user to use a hologram device to display and interact with a hologram of a deceased person.

[0710] "Means using an emotion engine" refers to technology that analyzes the user's voice and facial expressions to identify their current emotional state.

[0711] The "means for adjusting the hologram's actions and responses" is a function for appropriately changing the hologram's actions and speech based on the user's emotional state.

[0712] This invention is a system that uses data from the deceased's life to create a hologram using a generative AI model and combines it with an emotion engine to respond according to the user's emotional state. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[0713] Uploading data

[0714] Users access the system and upload data about the deceased. For example, a user accesses the system's website from a home computer or smartphone and uploads photos of their deceased grandfather or past messages. This uploaded data is sent to the server.

[0715] Receipt and confirmation of data

[0716] The server receives the uploaded data and checks its consistency and completeness. The server validates the uploaded file format (JPEG, PNG, etc.) and size (e.g., under 5MB), and ensures that images are loaded properly and audio files are not corrupted. This ensures that the data used by the system is available without any issues.

[0717] Data analysis

[0718] The server analyzes the received data and extracts the necessary information using the following techniques:

[0719] Natural language processing technology (NLP): Analyzes the deceased's frequently used phrases and speaking characteristics from uploaded text data.

[0720] Voice recognition technology: Analyzes voice data to determine the voice quality and speaking style of the deceased.

[0721] Image recognition technology: Analyzes the face and gestures of the deceased from photographs and videos.

[0722] For example, the server can understand the context of messages and letters from the deceased, identify common phrases and speaking characteristics, and even identify vocal characteristics from audio recordings.

[0723] Hologram generation

[0724] The AI ​​then generates a hologram of the deceased person based on the analyzed data, using the following techniques:

[0725] 3D modeling software (e.g. Blender): Create a 3D model of the deceased's appearance.

[0726] Speech synthesis software (e.g., Google Text-to-Speech): Generates an audio file and reproduces the deceased person's voice.

[0727] Motion capture technology (e.g., Kinect): Creating animations that replicate the movements of the deceased.

[0728] For example, generative AI can create a 3D model of your grandfather based on old photos, or generate natural-sounding conversational voices based on audio recordings.

[0729] Hologram data transfer

[0730] The server transfers the generated hologram data to the hologram device, packages the generated 3D model, audio file, and motion data, and sends the package to the hologram device. When the transfer is complete, the server records the completion of the data transfer.

[0731] User Notifications

[0732] The server notifies the user that the hologram data transfer has been completed. For example, the server sends a notification email to the user's terminal stating "Hologram data has been transferred to the device." It also displays a pop-up notification on the system's web page.

[0733] Hologram display

[0734] The user activates the hologram device and displays the hologram of the deceased. For example, the user turns on the hologram device, launches the application, and clicks the "Show hologram of the deceased" button, which displays the transferred hologram of the deceased.

[0735] Interaction and Emotion Recognition

[0736] Users can enjoy interacting with the hologram. By talking to the hologram or giving specific instructions, users can interact with the hologram and recreate memories of their deceased loved one. The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness."

[0737] Use of emotion engine

[0738] The hologram's behavior and responses are adjusted based on the identified emotional state. For example, if the user is sad, the hologram will provide a comforting message or a reminiscence. The emotion engine has the means to play pre-set memories and messages in response to specific emotional states. For example, if the user is moved, the hologram will deliver a warm message that corresponds to that emotion.

[0739] Prompt Sentence Examples

[0740] "Generate a hologram of the deceased person based on the following data: photographs, voice recordings, messages, letters, and design it to respond appropriately based on the user's emotional state."

[0741] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0742] Step 1:

[0743] The user accesses the system and uploads the deceased's data. The user accesses the system's website using a home computer or smartphone and clicks the "Upload" button. Next, on the upload screen, the user selects the deceased's audio, photos, videos, social media posts, words and actions, essays, letters, etc., and presses the "Send" button to send the data to the server. The input is the deceased's data file, and the output is the data uploaded to the server.

[0744] Step 2:

[0745] The server receives the uploaded data and checks its consistency and completeness. The server validates the received file format (JPEG, PNG, MP3, etc.) and size (e.g., less than 5MB), and ensures that images are loaded in the correct format and audio files are not corrupted. For example, it checks the header of a JPEG file to ensure that the image is valid. The input is the uploaded data file, and the output is the validated data file.

[0746] Step 3:

[0747] The server analyzes the received data and extracts the necessary information. Specifically, it uses natural language processing technology to analyze the text data and extract characteristics of the deceased's words and actions. It also uses speech recognition technology to analyze the audio data and extract the deceased's voice quality and speaking style, and image recognition technology to extract the deceased's appearance and movements from photos and videos. For example, it extracts frequently used phrases from the deceased's text messages. The input is verified data, and the output is analyzed data (text features, audio features, and image features).

[0748] Step 4:

[0749] The generative AI generates a hologram and voice model of the deceased based on the analyzed data. It creates a 3D model of the deceased using 3D modeling software (e.g., Blender), generates an audio file using speech synthesis software (e.g., Google Text-to-Speech), and creates animation data using motion capture technology (e.g., Kinect). The input is the analyzed data, and the output is the generated 3D model and audio file.

[0750] Step 5:

[0751] The server transfers the generated hologram data to the hologram device. The server then combines the generated 3D model, audio file, and motion data into a single package and transfers the package to the hologram device. For example, the server sends the data using the FTP protocol. The input is the generated hologram data, and the output is the data transferred to the hologram device.

[0752] Step 6:

[0753] The server notifies the user that the hologram data transfer is complete. The server sends a notification email to the user's email address stating "Hologram data has been transferred to the device" and also displays a pop-up notification on the system's web page. The input is the transfer completion status, and the output is the notification email and pop-up notification to the user.

[0754] Step 7:

[0755] The user turns on the hologram device and displays a hologram of the deceased. The user turns on the hologram device, launches the application, and clicks the "Show hologram of the deceased" button. The input is the hologram device, and the output is the displayed hologram of the deceased.

[0756] Step 8:

[0757] Users can enjoy interacting with holograms. By speaking to the hologram or giving specific instructions, users can interact with the hologram and recreate memories of their deceased loved ones. The input is the user's voice and actions, and the output is the hologram's response.

[0758] Step 9:

[0759] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness." The input is the user's voice and facial expression data, and the output is the identified emotional state.

[0760] Step 10:

[0761] The hologram's behavior and response are adjusted based on the identified emotional state. For example, if the user is sad, the hologram may offer comforting messages such as, "Are you OK? Is something wrong?" The input is the identified emotional state, and the output is the adjusted hologram's behavior and response.

[0762] (Application example 2)

[0763] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0764] In a hologram system based on data from the deceased's lifetime, when a user enjoys interacting with a hologram of the deceased, it is necessary to obtain appropriate responses according to the user's emotional state. In particular, when displaying holograms using smart glasses in a physical store, it is necessary to realize natural responses and interactions according to changes in the user's emotions.

[0765] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing text data using natural language processing technology and extracting characteristics of the deceased's words and actions; means for analyzing voice data using voice recognition technology and extracting the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology and extracting the deceased's appearance and movements; means for generating a hologram and voice model based on the deceased's data before death and the user's emotional state; means for transferring the generated hologram and voice model to a hologram device; means for the user to enjoy interaction with the deceased in a physical store using smart glasses; means for analyzing the user's voice and face data using an emotion engine and identifying the user's emotional state; and means for adjusting the hologram's behavior and response based on the identified emotional state. This makes it possible to achieve optimal responses and natural interactions according to the user's emotional state.

[0766] "Voice of the deceased" refers to voice data recorded by the deceased during their lifetime.

[0767] A "photograph" is a still image of the deceased or related objects or people.

[0768] "Video" is dynamic image data of the deceased or related objects or people.

[0769] "Social media posts" are content such as text, images, and videos posted by the deceased on social media while they were alive.

[0770] "Words and actions" is a record of the words and actions of the deceased during their lifetime.

[0771] "Compositions" refer to essays and writings written by the deceased during their lifetime.

[0772] A "letter" is a letter or message written by a deceased person while they were still alive.

[0773] "Means for uploading" refers to the procedures and interfaces that allow users to send data about the deceased to the server.

[0774] "Integrity and completeness check measures" are processes that verify that uploaded data is in the correct format and is complete.

[0775] "Natural language processing technology" is a technology for analyzing text data and extracting meaning and characteristics.

[0776] "Voice recognition technology" is a technology that analyzes voice data to identify the speaker's voice quality and speaking style.

[0777] "Image recognition technology" is a technology that analyzes images and videos to extract the appearance and movements of a target.

[0778] An "emotion engine" is a technology that identifies a user's emotional state from their voice and facial expressions and adjusts responses accordingly.

[0779] A "hologram device" is a device for displaying a generated hologram.

[0780] "Smart glasses" are augmented reality (AR) display devices worn by users.

[0781] "Means for enjoying interaction" are functions and processes that allow users to interact with holograms through dialogue and manipulation.

[0782] The system for carrying out the present invention includes the following processing flow and data analysis process.

[0783] 1. Upload your data

[0784] Users access the system using their home computers or smartphones and upload data such as audio, photos, videos, social media posts, words and actions, essays, and letters of the deceased, thereby collecting information about the deceased's activities and characteristics while they were alive.

[0785] 2. Receipt and confirmation of data

[0786] The server receives the data uploaded by the user and checks the consistency and completeness of each data item. This includes validating the file format and size, and testing the readability of image and audio files. For example, the server checks whether the received image file is corrupted.

[0787] 3. Data Analysis

[0788] The server analyzes the received data and extracts the required information using the following techniques:

[0789] Natural language processing technology (NLP): Analyzes the characteristics of the deceased's words and actions from text data.

[0790] Voice recognition technology: Analyzes the voice quality and speaking style of the deceased from audio data.

[0791] Image recognition technology: Analyzes the face and gestures of the deceased from photographs and videos.

[0792] 4. Hologram Generation

[0793] Based on the deceased's life data and the user's emotional state, the generative AI generates a hologram and voice model. Specifically, it uses 3D modeling technology to recreate the deceased's appearance, voice synthesis technology to generate the deceased's voice, and motion capture technology to recreate their movements and behavior.

[0794] 5. Transfer of hologram data

[0795] The server transfers the generated hologram data to the hologram device, which includes a package of 3D models, audio files, and motion data. After the transfer is complete, the server notifies the user of the transfer to their smart glasses.

[0796] 6. Interaction and Emotion Recognition

[0797] Users interact with holograms in brick-and-mortar stores using smart glasses. The smart glasses' cameras and microphones capture the user's voice and facial expressions and transmit them to a server. The server uses an emotion engine to analyze the user's emotional state and adjust the hologram's responses and behavior based on the identified emotion. For example, if the user has a sad expression, the hologram will convey a comforting message such as "Cheer up."

[0798] Examples of prompt statements

[0799] Prompt statement:

[0800] Recognize the user's emotional state and generate a hologram that responds based on that state. Take the following data as input:

[0801] 1. User's voice data: voice_sample.wav

[0802] 2. User's facial expression image: face_sample.png

[0803] 3. Text data of the deceased:grandfather_messages.txt

[0804] The generated hologram will have animation and audio that will convey an uplifting message if the user is feeling sad.

[0805] This allows users to recreate memories of the deceased in a physical store and receive appropriate responses based on their emotions.

[0806] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0807] Step 1:

[0808] Users access the system using their home computers or smartphones and upload data about the deceased, such as audio, photos, videos, social media posts, words and actions, essays, letters, etc. The input is various data related to the deceased, and the output is this data stored on the server.

[0809] Step 2:

[0810] The server receives the uploaded data and checks the consistency and completeness of each piece of data. The input is the file uploaded by the user, and the output is the verified data. Specifically, it validates the file format and size, and performs read tests on image and audio files. For example, the server checks whether the received image file is corrupted.

[0811] Step 3:

[0812] The server uses natural language processing technology to analyze text data and extract characteristics of the deceased's words and actions. The input is text data, and the output is data that represents the characteristics of the deceased's words and actions. Specifically, it uses an NLP library to perform context analysis and key phrase extraction.

[0813] Step 4:

[0814] The server uses voice recognition technology to analyze the voice data and extract the voice quality and speaking style of the deceased. The input is the voice data, and the output is characteristic data of the voice quality and speaking style of the deceased. Specifically, it uses a voice recognition library to analyze pitch and tone from the voice waveform.

[0815] Step 5:

[0816] The server uses image recognition technology to analyze photos and videos to extract the appearance and movements of the deceased. The input is photo and video data, and the output is data for recreating the appearance and movements of the deceased. Specific operations include facial recognition and motion capture using an image processing library.

[0817] Step 6:

[0818] The server uses generative AI to generate a hologram and voice model based on the deceased's data and the user's emotional state. The input is the analyzed characteristic data of the deceased and the user's emotional data, and the output is the hologram and voice model data. Specifically, the hologram is created using 3D modeling, voice synthesis, and motion capture technology.

[0819] Step 7:

[0820] The server packages the generated hologram data and transfers it to the hologram device. The input is the generated hologram and audio model data, and the output is data in a format that can be displayed on the hologram device. Specifically, data compression and transfer protocols are used to ensure rapid data transfer.

[0821] Step 8:

[0822] The user enjoys interacting with the deceased person in a physical store using smart glasses. The input is data transmitted to the hologram device, and the output is a displayed hologram. Specifically, the user launches the smart glasses application and begins interacting with the displayed hologram.

[0823] Step 9:

[0824] The user's voice and facial expression data are captured by the smart glasses and sent to the server. The input is the user's voice and facial expression data, and the output is trigger data to start the analysis. Specifically, the data is captured using the microphone and camera of the smart glasses and sent to the server via the network.

[0825] Step 10:

[0826] The server uses an emotion engine to analyze the user's voice and facial expression data to identify the user's emotional state. The input is the user's voice and facial expression data, and the output is the user's emotional state data. Specifically, the server applies voice analysis and facial expression analysis algorithms to analyze emotions.

[0827] Step 11:

[0828] The server adjusts the hologram's behavior and response based on the identified emotional state. The input is the user's emotional state data, and the output is the adjusted hologram response data. Specifically, it uses an AI model to generate and display emotionally appropriate hologram responses.

[0829] Examples of prompts:

[0830] Prompt statement:

[0831] Recognize the user's emotional state and generate a hologram that responds based on that state. Take the following data as input:

[0832] 1. User's voice data: voice_sample.wav

[0833] 2. User's facial expression image: face_sample.png

[0834] 3. Text data of the deceased:grandfather_messages.txt

[0835] The generated hologram will have animation and audio that will convey an uplifting message if the user is feeling sad.

[0836] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0837] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0838] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0839] [Third embodiment]

[0840] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0841] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0842] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0843] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0844] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0845] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0846] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0847] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0848] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0849] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0850] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0851] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0852] This invention is a system that uses generative AI to create holograms based on data from the deceased's life, providing comfort to surviving family members and loved ones. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[0853] Uploading data

[0854] The user accesses the system and uploads the necessary data on the deceased.

[0855] Example: A user accesses the system on their home computer or smartphone and uploads photos of their deceased grandfather and past messages.

[0856] Receipt and confirmation of data

[0857] The server receives the uploaded data and verifies its consistency and completeness.

[0858] For example: A server might validate uploaded file types and sizes, ensure images load properly, and ensure audio files aren't corrupted.

[0859] Data analysis

[0860] The server analyzes the received data and extracts the necessary information.

[0861] Using natural language processing (NLP) technology, the characteristics of the deceased's words and actions are analyzed from text data.

[0862] Using voice recognition technology, characteristic voice qualities and speaking styles are analyzed from audio data.

[0863] Image recognition technology is used to analyze the face and gestures of the deceased from photographs and videos.

[0864] Example: The server understands the context of messages and letters from the deceased and identifies common phrases and speaking characteristics.

[0865] Hologram generation

[0866] The generating AI generates a hologram of the deceased person based on the analyzed data.

[0867] Using 3D modeling technology, a 3D model is created that recreates the appearance of the deceased.

[0868] Using voice synthesis technology, an audio file is created that reproduces the voice of the deceased.

[0869] Motion capture technology is used to create animations that recreate the movements of the deceased.

[0870] Example: Generative AI creates a 3D model of your grandfather based on old photos and generates natural-sounding conversational voices based on audio recordings.

[0871] Hologram data transfer

[0872] The server transfers the generated hologram data to the hologram device.

[0873] The generated 3D model, audio file, and motion data are packaged into a single file and transferred to the hologram device.

[0874] Example: The server notifies the user that "the hologram of your grandfather is complete" and transfers the data to the device.

[0875] Hologram display

[0876] The user activates the hologram device, which displays a hologram of the deceased person.

[0877] Turn on the hologram device and start the specified application.

[0878] Example: A user turns on a hologram device in their living room and relives a holographic story their grandfather used to tell their family.

[0879] Interaction and Continued Use

[0880] Users enjoy interacting with the holograms.

[0881] Users can talk to the hologram and carry on a conversation.

[0882] New data analyses and topics are added to the hologram periodically.

[0883] Example: A user can ask a hologram, "What story would you like to tell us today?" and the hologram will relive a story from an old family photo.

[0884] As a result, this system can effectively utilize data from the deceased's lifetime, providing deep healing for family members and those involved.

[0885] The processing flow will be explained below.

[0886] Step 1:

[0887] Users access the system and upload data about the deceased. Specifically, they log in and open the data upload page. From there, they select files such as photos, audio, video, social media posts, essays, and letters, and press the upload button to send them to the server.

[0888] Step 2:

[0889] The server receives the uploaded data. Specifically, the server receives the data file sent by the user and saves it in a specified storage location.

[0890] Step 3:

[0891] The server checks the integrity and completeness of the data it receives. Specifically, it checks the file format, size, and data corruption status to check for any abnormalities. If an abnormality is found, it sends a notification to the user urging them to re-upload.

[0892] Step 4:

[0893] The server analyzes the data. Specifically, it uses natural language processing (NLP) technology to extract characteristics of the deceased's words and actions from the text data. It also uses a language model to analyze frequently occurring phrases and context.

[0894] Step 5:

[0895] The server uses voice recognition technology to analyze the audio data, extracting characteristics of the deceased's voice quality and speaking style from the audio data and storing these characteristics in a database.

[0896] Step 6:

[0897] The server uses image recognition technology to analyze photos and videos, extracting facial features and gestures of the deceased, and gathers the information needed to generate a 3D model.

[0898] Step 7:

[0899] Based on the analyzed data, the AI ​​generates a hologram of the deceased. Specifically, it uses 3D modeling technology to create a 3D model that recreates the appearance of the deceased, voice synthesis technology to generate an audio file that recreates the deceased's voice, and motion capture technology to create an animation that recreates the movements of the deceased.

[0900] Step 8:

[0901] The server transfers the generated hologram data to the hologram device. Specifically, it packages the generated 3D model, audio file, and motion data, and transfers the package to the hologram device.

[0902] Step 9:

[0903] The server notifies the user that the hologram data transfer has been completed. Specifically, the server notifies the user by sending a pop-up screen or email to the user's device stating that "the hologram data has been transferred to the device."

[0904] Step 10:

[0905] The user activates the hologram device and displays the hologram. Specifically, the user turns on the device, launches the application, and displays the transferred hologram of the deceased person.

[0906] Step 11:

[0907] Users can enjoy interacting with the hologram by speaking to it or giving it specific instructions, thereby recreating memories of the deceased.

[0908] Through the above processing steps, a hologram based on the data of the deceased person is generated, and the user can find emotional healing through it.

[0909] Example 1

[0910] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0911] In modern society, people deeply cherish memories of their deceased loved ones, but there are limited ways to physically experience them. Traditional memorial methods struggle to recreate the voices and movements of the deceased, resulting in the bereaved family members being unable to feel close to them. In particular, there is a need for a method to heal the hearts of bereaved family members by recreating the voices and movements of the deceased.

[0912] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0913] In this invention, the server includes: means for a user to upload data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for the server to verify the consistency and completeness of the uploaded data; means for the server to analyze the text data using natural language processing technology and extract characteristics of the deceased's words and actions; means for the server to analyze the voice data using voice recognition technology and extract the deceased's voice quality and speaking style; means for the server to analyze photos and videos using image recognition technology and extract the deceased's appearance and movements; means for the generation AI to generate a hologram and a voice model based on the generated characteristics of the deceased; means for the server to transfer the hologram and voice model to a hologram device; and means for the user to display a hologram of the deceased and interact with the deceased. This allows the user to interact with the hologram of the deceased and experience a physical sensation of the deceased.

[0914] "User" refers to an individual who accesses the system, uploads data of a deceased person, and uses a hologram device.

[0915] "Server" refers to the device or system that receives, verifies, and analyzes data uploaded by users, and generates and transmits holograms and audio models.

[0916] A "deceased person" refers to a person who has already passed away, but about whom the user wishes to retain memories.

[0917] "Audio data" refers to files containing recordings of the deceased person's voice.

[0918] "Photograph" refers to a still image that visually records the appearance of the deceased.

[0919] "Video" refers to a dynamic image file that includes the movements of the deceased.

[0920] "Social media posts" refers to data such as text, images, and videos posted by the deceased on social media platforms on the Internet.

[0921] "Words and actions" refers to the words and actions of the deceased.

[0922] "Composition" refers to a document written by the deceased.

[0923] A "letter" refers to a document written by a deceased person as a form of personal communication.

[0924] "Data Integrity" refers to the state in which uploaded data is accurate and complete.

[0925] "Natural language processing technology" refers to the technology of analyzing text data and extracting meaning from it.

[0926] "Voice recognition technology" refers to technology that analyzes voice data and extracts features from the voice.

[0927] "Image recognition technology" refers to the technology of analyzing images and videos and extracting features from them.

[0928] "Generative AI" refers to artificial intelligence technology that automatically generates holograms and audio models based on analyzed data.

[0929] "Hologram device" refers to a device that displays generated holograms and allows a user to interact with them.

[0930] "Interaction" refers to a two-way exchange between a user and a hologram.

[0931] The present invention is a system that uses AI to create holograms based on data from the deceased person's life, providing comfort to surviving family members and those involved. Specific embodiments for carrying out the present invention are described below.

[0932] Uploading data

[0933] The user accesses the system and uploads the necessary data about the deceased. This is accomplished by logging into the system's web portal using a home computer or smartphone and uploading photos, audio files, videos, text files, etc. of the deceased. For example, a user can select photos of their deceased grandfather and past messages to be imported into the system.

[0934] Receipt and confirmation of data

[0935] The server receives the uploaded data and checks its consistency and integrity. Specifically, it checks the file format (e.g., jpg, mp3, mp4, txt, etc.) of each file received by the server, and then calculates the file's hash value to ensure the data is not corrupted. If an improper file format or corruption is detected, an error message is returned to the user.

[0936] Data analysis

[0937] The server analyzes the received data and extracts the necessary information. Natural language processing (NLP) technology is used to analyze the text data and extract characteristics of the deceased's words and actions. For example, distinctive phrases and speaking characteristics are extracted from letters and emails uploaded by the user. Speech recognition technology is also used to analyze the audio data and identify the deceased's voice quality and intonation. Image recognition technology is used to analyze the deceased's face and movements from photos and videos. The specific software used at this stage is Python libraries (NLTK, SpaCy, OpenCV, etc.) and machine learning frameworks such as TensorFlow.

[0938] Hologram generation

[0939] The generative AI generates a hologram of the deceased based on the analyzed data. 3D modeling technology is used to create a 3D model that replicates the appearance of the deceased. Voice synthesis technology is used to create an audio file that replicates the deceased's voice. Motion capture technology is then used to create an animation that replicates the natural movements of the deceased. Specific software used at this stage includes Blender (3D modeling) and Vosk (voice synthesis). The prompt is something like, "Generate a 3D hologram of my grandfather based on his old photos and a recorded message."

[0940] Hologram data transfer

[0941] The server transfers the generated hologram data to the hologram device. The generated 3D model, audio file, and motion data are packaged and transferred to the hologram device as a single file. Once the transfer is complete, the user is notified.

[0942] Hologram display and interaction

[0943] The user activates the hologram device, which displays a hologram of the deceased. The user turns on the device and launches a specified application. The hologram device reads the data transferred from the server and displays a 3D hologram of the deceased. It plays audio and runs animations in real time. The user can talk to the hologram, which responds by reflecting the characteristics of the deceased. Enjoying this interaction brings back fond memories of the deceased and brings emotional healing.

[0944] In this way, the invention utilizes data from the deceased person's lifetime to provide a hologram that is extremely valuable to their family and friends.

[0945] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0946] Step 1:

[0947] Users access the system and upload data such as audio, photos, videos, social media posts, words, actions, essays, and letters of the deceased. The input is various data files (e.g., jpg, mp3, mp4, txt, etc.) owned by the user, and the output is the data files imported into the system. Specifically, users log in to the system's web portal using their home computer or smartphone and perform the upload operation.

[0948] Step 2:

[0949] The server receives data uploaded by users and checks its integrity and completeness. The input is the data file uploaded by the user, and the output is the data file whose integrity and completeness have been checked. Specifically, the server verifies the format of each file (e.g., jpg, mp3, mp4, etc.) and calculates a hash value to ensure the file is not corrupted. If the file is corrupted or in an improper format, an error message is returned to the user.

[0950] Step 3:

[0951] The server analyzes the received data and extracts the necessary information. The input is a data file whose consistency and completeness has been confirmed, and the output is a dataset with extracted characteristics of the deceased. Specific operations include using NLP technology to analyze the text data and extract commonly used phrases and speaking characteristics; analyzing the audio data using speech recognition technology to identify the voice quality and intonation of the deceased; and using image recognition technology to extract the face and movements of the deceased from photos and videos. Specific software used is Python libraries (NLTK, SpaCy, OpenCV, etc.) and TensorFlow.

[0952] Step 4:

[0953] The generation AI generates a hologram of the deceased based on the analyzed data. The input is the characteristic data of the deceased analyzed on the server, and the output is a hologram file containing a 3D model, audio file, and motion data. Specific operations involve using 3D modeling technology to create a 3D model that recreates the appearance of the deceased, and using voice synthesis technology to generate an audio file that recreates the deceased's voice. Furthermore, motion capture technology is used to create an animation that recreates the movements of the deceased. The specific software used is Blender (3D modeling) and Vosk (voice synthesis).

[0954] Step 5:

[0955] The server transfers the generated hologram data to the hologram device. The input is the generated hologram file, and the output is the data stored in the hologram device. Specifically, the server packages the generated 3D model, audio file, and motion data and transfers it to the hologram device as a single file. Once the transfer is complete, the server notifies the user that the data transfer is complete.

[0956] Step 6:

[0957] The user starts the hologram device and displays a hologram of the deceased. The input is a hologram file stored in the hologram device, and the output is a hologram of the deceased that is displayed and played back. Specifically, the user turns on the hologram device and launches a specified application. The hologram device reads the data, displays a 3D hologram of the deceased, plays audio in real time, and runs animations.

[0958] Step 7:

[0959] Users can enjoy interacting with the hologram. The input is responses from the hologram device and user input (voice and gestures), and the output is an interactive exchange experience. Specifically, when the user speaks to the hologram, the hologram responds using pre-analyzed data and real-time voice recognition technology. Continuing conversation and interaction with the user can revive fond memories of the deceased, bringing emotional healing.

[0960] (Application example 1)

[0961] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0962] Systems that more realistically recreate memories of the deceased and provide comfort to surviving family and friends face many technical challenges. In particular, systems that not only accurately reflect the characteristics of the deceased but also deepen communication with users through real interactions are required. Another challenge is to personalize the shopping experience and provide special value by recommending products based on the user's purchasing preferences.

[0963] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0964] In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing the text data using natural language processing technology to extract characteristics of the deceased's words and actions; means for analyzing the voice data using voice recognition technology to extract the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology to extract the deceased's appearance and movements; means for generating a hologram and a voice model based on the generated characteristics of the deceased; means for transferring the hologram and the voice model to a hologram device; means for displaying a hologram of the deceased and allowing the user to interact with the deceased; and means for recommending products based on the user's purchasing preferences. This enables the user's purchasing experience to be personalized while realistically recreating the deceased's memories.

[0965] A "deceased person" refers to a person who has passed away and is the person about whom data is to be reproduced.

[0966] "Data upload" refers to a user sending information about a deceased person (such as audio, photos, videos, social media posts, words, actions, writings, letters, etc.) to the system.

[0967] "Data integrity and completeness" means ensuring that uploaded data is accurate, complete, and not corrupted or missing.

[0968] "Natural language processing technology" is a technology that analyzes text data and extracts the characteristics and meaning of sentences and behavior.

[0969] "Voice recognition technology" is a technology that analyzes voice data and extracts characteristics such as speaking style and voice quality.

[0970] "Image recognition technology" is a technology that analyzes photographic and video data to extract a person's appearance and movements.

[0971] A "hologram" is a three-dimensional image reproduced to recreate the appearance and movements of a deceased person.

[0972] A "voice model" is digital voice data generated to reproduce the voice of a deceased person.

[0973] A "hologram device" is a hardware device for displaying generated holograms and audio models.

[0974] "Interaction" means that the user can have a conversation or dynamic interaction with the hologram of the deceased person.

[0975] "Purchasing preferences" refers to purchasing tendencies based on a user's interests and preferences.

[0976] "Product recommendation" refers to the process of presenting relevant products based on a user's purchasing preferences.

[0977] The system for implementing this invention generates a hologram and a voice model using various data of the deceased person, and allows the user to interact with the hologram and voice model and receive personalized product recommendations. Specific embodiments are described below.

[0978] System Overview

[0979] The system consists of the following main components:

[0980] 1. Data upload method

[0981] 2. Means of verifying data consistency and completeness

[0982] 3. Data analysis methods (natural language processing technology, voice recognition technology, image recognition technology)

[0983] 4. Hologram and audio model generation method

[0984] 5. Transfer method to hologram device

[0985] 6. Holographic Display and Interaction Methods

[0986] 7. Product recommendation means

[0987] Hardware and software used

[0988] Hardware:

[0989] Common devices such as smartphones, tablets, and personal computers

[0990] Dedicated hologram device

[0991] software:

[0992] Server Application

[0993] Natural Language Processing (NLP) Engine

[0994] Speech Recognition Engine

[0995] Image Recognition Engine

[0996] Generative AI Models

[0997] Speech synthesis engine

[0998] Product recommendation engine

[0999] Operation flow

[1000] 1. Data upload:

[1001] Users use their own devices (such as smartphones) to upload data such as audio, photos, videos, social media posts, essays, and letters of the deceased to the system.

[1002] 2. Data consistency and integrity checks:

[1003] The server checks the integrity and completeness of the uploaded data, specifically by validating the file format, size, and content, and prompting the user to re-upload if there are any issues.

[1004] 3. Data Analysis:

[1005] The server will analyze the uploaded data using the following analysis techniques:

[1006] Text analysis: Using NLP techniques to extract characteristics of the deceased's words and actions from text data.

[1007] Speech analysis: Using speech recognition technology to extract voice quality and speaking style from audio data.

[1008] Image analysis: Using image recognition technology to analyze the appearance and movements of the deceased from photographs and videos.

[1009] 4. Hologram and audio model generation:

[1010] A generative AI model is used to generate a hologram and voice model of the deceased person based on the acquired feature data.

[1011] 5. Transfer to hologram device:

[1012] The generated hologram and audio model are packaged and transferred to the hologram device.

[1013] 6. Hologram Display and Interaction:

[1014] The user activates the hologram device and interacts with the hologram of the deceased person. The hologram responds to the user's input, realizing a dialogue with the user.

[1015] 7. Product recommendation:

[1016] Using a product recommendation engine, the hologram suggests products based on the user's purchasing preferences, just like a shop assistant, and explains the reasons and features of the recommended products.

[1017] Prompt Sentence Examples

[1018] Examples:

[1019] User: The user opens a shopping app on their smartphone and enjoys shopping while viewing a hologram of their deceased mother. When the user says, "I want a new shirt," the hologram suggests several shirts and explains why it recommends them.

[1020] Example prompt sentence:

[1021] text

[1022] Show me a hologram of a deceased person who knows my preferences and suggests products in a specific category (e.g., "shirts"), how I chose them, and why I recommend them.

[1023] Using this system, users can not only recreate memories of their deceased loved ones, but also have a personalized shopping experience through interactions with the deceased.

[1024] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1025] Step 1:

[1026] A user uses a device (smartphone or tablet) to upload data about the deceased (audio, photos, videos, social media posts, words, actions, essays, letters, etc.) to the system. These files are sent to the server as input data. The server temporarily stores the received data as output.

[1027] Step 2:

[1028] The server verifies the integrity and completeness of the uploaded data. As input, it validates the uploaded data file, checking the file format, size, whether it is corrupted, and ensuring that all data has been received correctly. As output, it generates a flag indicating whether the data is valid.

[1029] Step 3:

[1030] The server analyzes the received data. The input data is the deceased person's data received in step 1. Specifically, the following process is performed:

[1031] Using natural language processing technology, characteristics of the deceased's words and actions are extracted from text data.

[1032] Using voice recognition technology, voice quality and speaking style are extracted from audio data.

[1033] Using image recognition technology, the appearance and movements of the deceased are extracted from photographs and videos.

[1034] As an output, feature data of the deceased extracted by each technique is generated.

[1035] Step 4:

[1036] A generative AI model is used to generate a hologram and voice model based on the analyzed feature data. The feature data obtained in step 3 is used as input. Specifically, the appearance of the deceased is recreated using 3D modeling software, and the voice is recreated using voice synthesis technology. The output is hologram data and voice data.

[1037] Step 5:

[1038] The server transfers the generated hologram and audio data to the hologram device. As input, the data generated in step 4 is used. As output, it is sent to the hologram device, where it is converted into a displayable format.

[1039] Step 6:

[1040] The user activates the hologram device and displays a hologram of the deceased person. As input, hologram and audio data are loaded into the hologram device. Specifically, the user turns on the device and launches a dedicated application. As output, a hologram of the deceased person is displayed and interaction begins.

[1041] Step 7:

[1042] The hologram recommends products based on the user's purchasing preferences. The user's purchasing history and preference data are used as input. Specifically, the product recommendation engine analyzes the user's preferences and generates a product list based on them. As output, the hologram explains to the user the recommended products, their features, and the reasons for the recommendation.

[1043] Prompt Sentence Examples

[1044] Show me a hologram of a deceased person who knows my preferences and suggests products in a specific category (e.g., "shirts"), how I chose them, and why I recommend them.

[1045] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1046] This invention is a system that uses generative AI to create a hologram based on data from the deceased's life, and combines it with an emotion engine to respond according to the user's emotional state. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[1047] Uploading data

[1048] A user accesses the system and uploads data about the deceased. For example, a user accesses the system on a home computer or smartphone and uploads photos of a deceased grandfather or past messages.

[1049] Receipt and confirmation of data

[1050] The server receives the uploaded data and checks its consistency and completeness. For example, the server validates the uploaded file format and size, ensures that images load properly, and that audio files are not corrupted.

[1051] Data analysis

[1052] The server analyzes the received data and extracts the necessary information.

[1053] Using natural language processing (NLP) technology, the characteristics of the deceased's words and actions are analyzed from text data.

[1054] Using voice recognition technology, characteristic voice qualities and speaking styles are analyzed from audio data.

[1055] Using image recognition technology, the server analyzes the face and body language of the deceased from photos and videos. For example, the server can understand the context of messages and letters from the deceased and identify common phrases and speaking characteristics.

[1056] Hologram generation

[1057] The generating AI generates a hologram of the deceased person based on the analyzed data.

[1058] 3D modeling technology is used to create a 3D model that recreates the appearance of the deceased.

[1059] Voice synthesis technology is used to create an audio file that reproduces the voice of the deceased.

[1060] Motion capture technology is used to create animations that replicate the movements of the deceased, while generative AI creates a 3D model of your grandfather based on old photos, and generates natural-sounding conversational voices based on audio recordings.

[1061] Hologram data transfer

[1062] The server transfers the generated hologram data to the hologram device, packages the generated 3D model, audio file, and motion data, and transfers the package to the hologram device.

[1063] User Notifications

[1064] The server notifies the user that the hologram data transfer has been completed. For example, the server notifies the user by sending a message to the user's device via a pop-up screen or email saying "The hologram data has been transferred to your device."

[1065] Hologram display

[1066] The user activates the hologram device and displays the hologram of the deceased person, e.g., by turning on the device, launching an application, and displaying the transferred hologram of the deceased person.

[1067] Interaction and Emotion Recognition

[1068] Users can enjoy interacting with the holograms. By speaking to the holograms or giving them specific instructions, users can interact with the holograms and relive memories of their deceased loved ones.

[1069] Use of emotion engine

[1070] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness."

[1071] The hologram's behavior and responses are tailored based on the identified emotional state: for example, if the user is sad, the hologram will offer a comforting message or a reminiscence.

[1072] The emotion engine has a means to play pre-set memories and messages according to a specific emotional state. For example, if the user is emotional, the hologram will convey a warm message according to that emotion.

[1073] As a result, this system effectively utilizes data from the deceased person's lifetime and provides optimal responses according to the user's emotional state, thereby achieving deep healing.

[1074] The processing flow will be explained below.

[1075] Step 1:

[1076] Users access the system and upload data about the deceased. For example, they log in to a dedicated web application on their home computer or smartphone, select files such as photos, audio, video, social media posts, essays, and letters of the deceased, and press the upload button to send them to the server.

[1077] Step 2:

[1078] The server receives the uploaded data and checks its integrity and completeness. Specifically, it checks the file format and size of the data received, ensures that images are loaded properly, and that audio files are not corrupted. If there are any abnormalities, it sends a notification to the user, encouraging them to re-upload.

[1079] Step 3:

[1080] The server analyzes the received data. Specifically,

[1081] Using natural language processing (NLP) technology, characteristics of the deceased's words and actions are extracted from text data (letters, social media posts, etc.).

[1082] Voice recognition technology is used to extract the voice quality and speaking style of the deceased from audio data (such as recordings).

[1083] Image recognition technology is used to extract facial and movement characteristics of the deceased from photographs and videos.

[1084] Step 4:

[1085] The AI ​​generates a hologram of the deceased person based on the analyzed data.

[1086] Using 3D modeling technology, a 3D model is created that recreates the appearance of the deceased based on collected facial and body features.

[1087] Using voice synthesis technology, an audio file is created that reproduces the voice of the deceased based on the extracted voice quality and speaking characteristics.

[1088] Motion capture technology is used to create animations that recreate the movements of the deceased.

[1089] Step 5:

[1090] The server transfers the generated hologram data to the hologram device. Specifically, it combines the generated 3D model, audio file, and motion data into a single package and sends it to the hologram device.

[1091] Step 6:

[1092] The server notifies the user that the hologram data transfer has been completed. Specifically, the server sends a pop-up notification or email to the user's device stating, "Hologram data has been transferred to your device."

[1093] Step 7:

[1094] The user activates the hologram device and displays the hologram of the deceased. Specifically, the user turns on the hologram device, launches the dedicated application, reads the transferred hologram data, and displays the hologram of the deceased.

[1095] Step 8:

[1096] Users can enjoy interacting with holograms. By talking to the hologram or giving specific instructions, the hologram will respond, establishing a dialogue.

[1097] Step 9:

[1098] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. Specifically, the hologram device captures the user's facial expressions and tone of voice through a built-in camera and microphone, and the emotion engine analyzes the data to determine emotions such as "sadness," "joy," and "surprise."

[1099] Step 10:

[1100] The server adjusts the hologram's behavior and responses based on the identified emotional state. For example, if the user is sad, the hologram will offer comforting words or warm messages, including reminiscences. If the user is happy, the hologram will engage in proactive interactions tailored to that emotion.

[1101] Step 11:

[1102] The server plays pre-set memories and messages according to the emotional state identified by the emotion engine. For example, if the user is emotional, a special message corresponding to that emotion will be played through the hologram.

[1103] Through the above specific processing steps, the system effectively utilizes data from the deceased person's lifetime and provides appropriate responses according to the user's emotional state, thereby achieving emotional healing.

[1104] Example 2

[1105] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1106] In today's world, there is a need for technology that can preserve the memory and presence of the deceased. However, conventional methods have difficulty reproducing the detailed characteristics of the deceased, and they are unable to respond to the user's emotional state. Therefore, there is a need for technology that allows users to feel deep healing through interactions with the deceased.

[1107] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing text data using natural language processing technology to extract characteristics of the deceased's words and actions; means for analyzing voice data using voice recognition technology to extract the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology to extract the deceased's appearance and movements; means for generating a hologram and voice model of the deceased using a generative AI based on the analyzed data; means for transferring the generated hologram and voice model to a hologram device; means for displaying a hologram of the deceased and allowing the user to interact with the deceased; means for analyzing the user's voice and facial expression data and using an emotion engine to identify the user's emotional state; and means for adjusting the hologram's behavior and responses based on the identified emotional state. This enables a high level of reproduction of the deceased's memories and enables appropriate responses tailored to the user's emotional state.

[1108] "Means for uploading data" means the ability for a user to access the system and send data to the system, such as audio, photographs, videos, social media posts, statements, writings, letters, etc., of the deceased.

[1109] "Means for verifying integrity and completeness" refers to the functionality by which the server verifies that data uploaded to the system is in the proper format and is not corrupted.

[1110] "Means of analyzing text data using natural language processing technology and extracting characteristics of speech and behavior" refers to technology for analyzing phrases and speaking styles frequently used by the deceased from uploaded text data.

[1111] "Means for analyzing audio data and extracting voice quality and speaking style" refers to technology that analyzes uploaded audio data and identifies the tone of the deceased's voice and speaking style characteristics.

[1112] "Means of analyzing images and videos to extract appearance and movements" refers to technology for detecting the facial features and physical movements of the deceased from uploaded photos and videos.

[1113] "Means for generating holograms and audio models using generative AI" refers to technology for generating 3D models and audio files of deceased people based on analyzed data.

[1114] The "means for transferring the hologram and audio model to the hologram device" is a function for packaging the generated data and sending it to the device that displays the hologram.

[1115] "Means for displaying and interacting with holograms" refers to a function that allows a user to use a hologram device to display and interact with a hologram of a deceased person.

[1116] "Means using an emotion engine" refers to technology that analyzes the user's voice and facial expressions to identify their current emotional state.

[1117] The "means for adjusting the hologram's actions and responses" is a function for appropriately changing the hologram's actions and speech based on the user's emotional state.

[1118] This invention is a system that uses data from the deceased's life to create a hologram using a generative AI model and combines it with an emotion engine to respond according to the user's emotional state. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[1119] Uploading data

[1120] Users access the system and upload data about the deceased. For example, a user accesses the system's website from a home computer or smartphone and uploads photos of their deceased grandfather or past messages. This uploaded data is sent to the server.

[1121] Receipt and confirmation of data

[1122] The server receives the uploaded data and checks its consistency and completeness. The server validates the uploaded file format (JPEG, PNG, etc.) and size (e.g., under 5MB), and ensures that images are loaded properly and audio files are not corrupted. This ensures that the data used by the system is available without any issues.

[1123] Data analysis

[1124] The server analyzes the received data and extracts the necessary information using the following techniques:

[1125] Natural language processing technology (NLP): Analyzes the deceased's frequently used phrases and speaking characteristics from uploaded text data.

[1126] Voice recognition technology: Analyzes voice data to determine the voice quality and speaking style of the deceased.

[1127] Image recognition technology: Analyzes the face and gestures of the deceased from photographs and videos.

[1128] For example, the server can understand the context of messages and letters from the deceased, identify common phrases and speaking characteristics, and even identify vocal characteristics from audio recordings.

[1129] Hologram generation

[1130] The AI ​​then generates a hologram of the deceased person based on the analyzed data, using the following techniques:

[1131] 3D modeling software (e.g. Blender): Create a 3D model of the deceased's appearance.

[1132] Speech synthesis software (e.g., Google Text-to-Speech): Generates an audio file and reproduces the deceased person's voice.

[1133] Motion capture technology (e.g., Kinect): Creating animations that replicate the movements of the deceased.

[1134] For example, generative AI can create a 3D model of your grandfather based on old photos, or generate natural-sounding conversational voices based on audio recordings.

[1135] Hologram data transfer

[1136] The server transfers the generated hologram data to the hologram device, packages the generated 3D model, audio file, and motion data, and sends the package to the hologram device. When the transfer is complete, the server records the completion of the data transfer.

[1137] User Notifications

[1138] The server notifies the user that the hologram data transfer has been completed. For example, the server sends a notification email to the user's terminal stating "Hologram data has been transferred to the device." It also displays a pop-up notification on the system's web page.

[1139] Hologram display

[1140] The user activates the hologram device and displays the hologram of the deceased. For example, the user turns on the hologram device, launches the application, and clicks the "Show hologram of the deceased" button, which displays the transferred hologram of the deceased.

[1141] Interaction and Emotion Recognition

[1142] Users can enjoy interacting with the hologram. By talking to the hologram or giving specific instructions, users can interact with the hologram and recreate memories of their deceased loved one. The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness."

[1143] Use of emotion engine

[1144] The hologram's behavior and responses are adjusted based on the identified emotional state. For example, if the user is sad, the hologram will provide a comforting message or a reminiscence. The emotion engine has the means to play pre-set memories and messages in response to specific emotional states. For example, if the user is moved, the hologram will deliver a warm message that corresponds to that emotion.

[1145] Prompt Sentence Examples

[1146] "Generate a hologram of the deceased person based on the following data: photographs, voice recordings, messages, letters, and design it to respond appropriately based on the user's emotional state."

[1147] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1148] Step 1:

[1149] The user accesses the system and uploads the deceased's data. The user accesses the system's website using a home computer or smartphone and clicks the "Upload" button. Next, on the upload screen, the user selects the deceased's audio, photos, videos, social media posts, words and actions, essays, letters, etc., and presses the "Send" button to send the data to the server. The input is the deceased's data file, and the output is the data uploaded to the server.

[1150] Step 2:

[1151] The server receives the uploaded data and checks its consistency and completeness. The server validates the received file format (JPEG, PNG, MP3, etc.) and size (e.g., less than 5MB), and ensures that images are loaded in the correct format and audio files are not corrupted. For example, it checks the header of a JPEG file to ensure that the image is valid. The input is the uploaded data file, and the output is the validated data file.

[1152] Step 3:

[1153] The server analyzes the received data and extracts the necessary information. Specifically, it uses natural language processing technology to analyze the text data and extract characteristics of the deceased's words and actions. It also uses speech recognition technology to analyze the audio data and extract the deceased's voice quality and speaking style, and image recognition technology to extract the deceased's appearance and movements from photos and videos. For example, it extracts frequently used phrases from the deceased's text messages. The input is verified data, and the output is analyzed data (text features, audio features, and image features).

[1154] Step 4:

[1155] The generative AI generates a hologram and voice model of the deceased based on the analyzed data. It creates a 3D model of the deceased using 3D modeling software (e.g., Blender), generates an audio file using speech synthesis software (e.g., Google Text-to-Speech), and creates animation data using motion capture technology (e.g., Kinect). The input is the analyzed data, and the output is the generated 3D model and audio file.

[1156] Step 5:

[1157] The server transfers the generated hologram data to the hologram device. The server then combines the generated 3D model, audio file, and motion data into a single package and transfers the package to the hologram device. For example, the server sends the data using the FTP protocol. The input is the generated hologram data, and the output is the data transferred to the hologram device.

[1158] Step 6:

[1159] The server notifies the user that the hologram data transfer is complete. The server sends a notification email to the user's email address stating "Hologram data has been transferred to the device" and also displays a pop-up notification on the system's web page. The input is the transfer completion status, and the output is the notification email and pop-up notification to the user.

[1160] Step 7:

[1161] The user turns on the hologram device and displays a hologram of the deceased. The user turns on the hologram device, launches the application, and clicks the "Show hologram of the deceased" button. The input is the hologram device, and the output is the displayed hologram of the deceased.

[1162] Step 8:

[1163] Users can enjoy interacting with holograms. By speaking to the hologram or giving specific instructions, users can interact with the hologram and recreate memories of their deceased loved ones. The input is the user's voice and actions, and the output is the hologram's response.

[1164] Step 9:

[1165] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness." The input is the user's voice and facial expression data, and the output is the identified emotional state.

[1166] Step 10:

[1167] The hologram's behavior and response are adjusted based on the identified emotional state. For example, if the user is sad, the hologram may offer comforting messages such as, "Are you OK? Is something wrong?" The input is the identified emotional state, and the output is the adjusted hologram's behavior and response.

[1168] (Application example 2)

[1169] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1170] In a hologram system based on data from the deceased's lifetime, when a user enjoys interacting with a hologram of the deceased, it is necessary to obtain appropriate responses according to the user's emotional state. In particular, when displaying holograms using smart glasses in a physical store, it is necessary to realize natural responses and interactions according to changes in the user's emotions.

[1171] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing text data using natural language processing technology and extracting characteristics of the deceased's words and actions; means for analyzing voice data using voice recognition technology and extracting the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology and extracting the deceased's appearance and movements; means for generating a hologram and voice model based on the deceased's data before death and the user's emotional state; means for transferring the generated hologram and voice model to a hologram device; means for the user to enjoy interaction with the deceased in a physical store using smart glasses; means for analyzing the user's voice and face data using an emotion engine and identifying the user's emotional state; and means for adjusting the hologram's behavior and response based on the identified emotional state. This makes it possible to achieve optimal responses and natural interactions according to the user's emotional state.

[1172] "Voice of the deceased" refers to voice data recorded by the deceased during their lifetime.

[1173] A "photograph" is a still image of the deceased or related objects or people.

[1174] "Video" is dynamic image data of the deceased or related objects or people.

[1175] "Social media posts" are content such as text, images, and videos posted by the deceased on social media while they were alive.

[1176] "Words and actions" is a record of the words and actions of the deceased during their lifetime.

[1177] "Compositions" refer to essays and writings written by the deceased during their lifetime.

[1178] A "letter" is a letter or message written by a deceased person while they were still alive.

[1179] "Means for uploading" refers to the procedures and interfaces that allow users to send data about the deceased to the server.

[1180] "Integrity and completeness check measures" are processes that verify that uploaded data is in the correct format and is complete.

[1181] "Natural language processing technology" is a technology for analyzing text data and extracting meaning and characteristics.

[1182] "Voice recognition technology" is a technology that analyzes voice data to identify the speaker's voice quality and speaking style.

[1183] "Image recognition technology" is a technology that analyzes images and videos to extract the appearance and movements of a target.

[1184] An "emotion engine" is a technology that identifies a user's emotional state from their voice and facial expressions and adjusts responses accordingly.

[1185] A "hologram device" is a device for displaying a generated hologram.

[1186] "Smart glasses" are augmented reality (AR) display devices worn by users.

[1187] "Means for enjoying interaction" are functions and processes that allow users to interact with holograms through dialogue and manipulation.

[1188] The system for carrying out the present invention includes the following processing flow and data analysis process.

[1189] 1. Upload your data

[1190] Users access the system using their home computers or smartphones and upload data such as audio, photos, videos, social media posts, words and actions, essays, and letters of the deceased, thereby collecting information about the deceased's activities and characteristics while they were alive.

[1191] 2. Receipt and confirmation of data

[1192] The server receives the data uploaded by the user and checks the consistency and completeness of each data item. This includes validating the file format and size, and testing the readability of image and audio files. For example, the server checks whether the received image file is corrupted.

[1193] 3. Data Analysis

[1194] The server analyzes the received data and extracts the required information using the following techniques:

[1195] Natural language processing technology (NLP): Analyzes the characteristics of the deceased's words and actions from text data.

[1196] Voice recognition technology: Analyzes the voice quality and speaking style of the deceased from audio data.

[1197] Image recognition technology: Analyzes the face and gestures of the deceased from photographs and videos.

[1198] 4. Hologram Generation

[1199] Based on the deceased's life data and the user's emotional state, the generative AI generates a hologram and voice model. Specifically, it uses 3D modeling technology to recreate the deceased's appearance, voice synthesis technology to generate the deceased's voice, and motion capture technology to recreate their movements and behavior.

[1200] 5. Transfer of hologram data

[1201] The server transfers the generated hologram data to the hologram device, which includes a package of 3D models, audio files, and motion data. After the transfer is complete, the server notifies the user of the transfer to their smart glasses.

[1202] 6. Interaction and Emotion Recognition

[1203] Users interact with holograms in brick-and-mortar stores using smart glasses. The smart glasses' cameras and microphones capture the user's voice and facial expressions and transmit them to a server. The server uses an emotion engine to analyze the user's emotional state and adjust the hologram's responses and behavior based on the identified emotion. For example, if the user has a sad expression, the hologram will convey a comforting message such as "Cheer up."

[1204] Examples of prompt statements

[1205] Prompt statement:

[1206] Recognize the user's emotional state and generate a hologram that responds based on that state. Take the following data as input:

[1207] 1. User's voice data: voice_sample.wav

[1208] 2. User's facial expression image: face_sample.png

[1209] 3. Text data of the deceased:grandfather_messages.txt

[1210] The generated hologram will have animation and audio that will convey an uplifting message if the user is feeling sad.

[1211] This allows users to recreate memories of the deceased in a physical store and receive appropriate responses based on their emotions.

[1212] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1213] Step 1:

[1214] Users access the system using their home computers or smartphones and upload data about the deceased, such as audio, photos, videos, social media posts, words and actions, essays, letters, etc. The input is various data related to the deceased, and the output is this data stored on the server.

[1215] Step 2:

[1216] The server receives the uploaded data and checks the consistency and completeness of each piece of data. The input is the file uploaded by the user, and the output is the verified data. Specifically, it validates the file format and size, and performs read tests on image and audio files. For example, the server checks whether the received image file is corrupted.

[1217] Step 3:

[1218] The server uses natural language processing technology to analyze text data and extract characteristics of the deceased's words and actions. The input is text data, and the output is data that represents the characteristics of the deceased's words and actions. Specifically, it uses an NLP library to perform context analysis and key phrase extraction.

[1219] Step 4:

[1220] The server uses voice recognition technology to analyze the voice data and extract the voice quality and speaking style of the deceased. The input is the voice data, and the output is characteristic data of the voice quality and speaking style of the deceased. Specifically, it uses a voice recognition library to analyze pitch and tone from the voice waveform.

[1221] Step 5:

[1222] The server uses image recognition technology to analyze photos and videos to extract the appearance and movements of the deceased. The input is photo and video data, and the output is data for recreating the appearance and movements of the deceased. Specific operations include facial recognition and motion capture using an image processing library.

[1223] Step 6:

[1224] The server uses generative AI to generate a hologram and voice model based on the deceased's data and the user's emotional state. The input is the analyzed characteristic data of the deceased and the user's emotional data, and the output is the hologram and voice model data. Specifically, the hologram is created using 3D modeling, voice synthesis, and motion capture technology.

[1225] Step 7:

[1226] The server packages the generated hologram data and transfers it to the hologram device. The input is the generated hologram and audio model data, and the output is data in a format that can be displayed on the hologram device. Specifically, data compression and transfer protocols are used to ensure rapid data transfer.

[1227] Step 8:

[1228] The user enjoys interacting with the deceased person in a physical store using smart glasses. The input is data transmitted to the hologram device, and the output is a displayed hologram. Specifically, the user launches the smart glasses application and begins interacting with the displayed hologram.

[1229] Step 9:

[1230] The user's voice and facial expression data are captured by the smart glasses and sent to the server. The input is the user's voice and facial expression data, and the output is trigger data to start the analysis. Specifically, the data is captured using the microphone and camera of the smart glasses and sent to the server via the network.

[1231] Step 10:

[1232] The server uses an emotion engine to analyze the user's voice and facial expression data to identify the user's emotional state. The input is the user's voice and facial expression data, and the output is the user's emotional state data. Specifically, the server applies voice analysis and facial expression analysis algorithms to analyze emotions.

[1233] Step 11:

[1234] The server adjusts the hologram's behavior and response based on the identified emotional state. The input is the user's emotional state data, and the output is the adjusted hologram response data. Specifically, it uses an AI model to generate and display emotionally appropriate hologram responses.

[1235] Examples of prompts:

[1236] Prompt statement:

[1237] Recognize the user's emotional state and generate a hologram that responds based on that state. Take the following data as input:

[1238] 1. User's voice data: voice_sample.wav

[1239] 2. User's facial expression image: face_sample.png

[1240] 3. Text data of the deceased:grandfather_messages.txt

[1241] The generated hologram will have animation and audio that will convey an uplifting message if the user is feeling sad.

[1242] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1243] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1244] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1245] [Fourth embodiment]

[1246] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1247] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1248] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1249] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1250] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1251] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1252] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1253] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1254] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1255] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1256] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1257] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1258] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1259] This invention is a system that uses generative AI to create holograms based on data from the deceased's life, providing comfort to surviving family members and loved ones. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[1260] Uploading data

[1261] The user accesses the system and uploads the necessary data on the deceased.

[1262] Example: A user accesses the system on their home computer or smartphone and uploads photos of their deceased grandfather and past messages.

[1263] Receipt and confirmation of data

[1264] The server receives the uploaded data and verifies its consistency and completeness.

[1265] For example: A server might validate uploaded file types and sizes, ensure images load properly, and ensure audio files aren't corrupted.

[1266] Data analysis

[1267] The server analyzes the received data and extracts the necessary information.

[1268] Using natural language processing (NLP) technology, the characteristics of the deceased's words and actions are analyzed from text data.

[1269] Using voice recognition technology, characteristic voice qualities and speaking styles are analyzed from audio data.

[1270] Image recognition technology is used to analyze the face and gestures of the deceased from photographs and videos.

[1271] Example: The server understands the context of messages and letters from the deceased and identifies common phrases and speaking characteristics.

[1272] Hologram generation

[1273] The generating AI generates a hologram of the deceased person based on the analyzed data.

[1274] Using 3D modeling technology, a 3D model is created that recreates the appearance of the deceased.

[1275] Using voice synthesis technology, an audio file is created that reproduces the voice of the deceased.

[1276] Motion capture technology is used to create animations that recreate the movements of the deceased.

[1277] Example: Generative AI creates a 3D model of your grandfather based on old photos and generates natural-sounding conversational voices based on audio recordings.

[1278] Hologram data transfer

[1279] The server transfers the generated hologram data to the hologram device.

[1280] The generated 3D model, audio file, and motion data are packaged into a single file and transferred to the hologram device.

[1281] Example: The server notifies the user that "the hologram of your grandfather is complete" and transfers the data to the device.

[1282] Hologram display

[1283] The user activates the hologram device, which displays a hologram of the deceased person.

[1284] Turn on the hologram device and start the specified application.

[1285] Example: A user turns on a hologram device in their living room and relives a holographic story their grandfather used to tell their family.

[1286] Interaction and Continued Use

[1287] Users enjoy interacting with the holograms.

[1288] Users can talk to the hologram and carry on a conversation.

[1289] New data analyses and topics are added to the hologram periodically.

[1290] Example: A user can ask a hologram, "What story would you like to tell us today?" and the hologram will relive a story from an old family photo.

[1291] As a result, this system can effectively utilize data from the deceased's lifetime, providing deep healing for family members and those involved.

[1292] The processing flow will be explained below.

[1293] Step 1:

[1294] Users access the system and upload data about the deceased. Specifically, they log in and open the data upload page. From there, they select files such as photos, audio, video, social media posts, essays, and letters, and press the upload button to send them to the server.

[1295] Step 2:

[1296] The server receives the uploaded data. Specifically, the server receives the data file sent by the user and saves it in a specified storage location.

[1297] Step 3:

[1298] The server checks the integrity and completeness of the data it receives. Specifically, it checks the file format, size, and data corruption status to check for any abnormalities. If an abnormality is found, it sends a notification to the user urging them to re-upload.

[1299] Step 4:

[1300] The server analyzes the data. Specifically, it uses natural language processing (NLP) technology to extract characteristics of the deceased's words and actions from the text data. It also uses a language model to analyze frequently occurring phrases and context.

[1301] Step 5:

[1302] The server uses voice recognition technology to analyze the audio data, extracting characteristics of the deceased's voice quality and speaking style from the audio data and storing these characteristics in a database.

[1303] Step 6:

[1304] The server uses image recognition technology to analyze photos and videos, extracting facial features and gestures of the deceased, and gathers the information needed to generate a 3D model.

[1305] Step 7:

[1306] Based on the analyzed data, the AI ​​generates a hologram of the deceased. Specifically, it uses 3D modeling technology to create a 3D model that recreates the appearance of the deceased, voice synthesis technology to generate an audio file that recreates the deceased's voice, and motion capture technology to create an animation that recreates the movements of the deceased.

[1307] Step 8:

[1308] The server transfers the generated hologram data to the hologram device. Specifically, it packages the generated 3D model, audio file, and motion data, and transfers the package to the hologram device.

[1309] Step 9:

[1310] The server notifies the user that the hologram data transfer has been completed. Specifically, the server notifies the user by sending a pop-up screen or email to the user's device stating that "the hologram data has been transferred to the device."

[1311] Step 10:

[1312] The user activates the hologram device and displays the hologram. Specifically, the user turns on the device, launches the application, and displays the transferred hologram of the deceased person.

[1313] Step 11:

[1314] Users can enjoy interacting with the hologram by speaking to it or giving it specific instructions, thereby recreating memories of the deceased.

[1315] Through the above processing steps, a hologram based on the data of the deceased person is generated, and the user can find emotional healing through it.

[1316] Example 1

[1317] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1318] In modern society, people deeply cherish memories of their deceased loved ones, but there are limited ways to physically experience them. Traditional memorial methods struggle to recreate the voices and movements of the deceased, resulting in the bereaved family members being unable to feel close to them. In particular, there is a need for a method to heal the hearts of bereaved family members by recreating the voices and movements of the deceased.

[1319] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1320] In this invention, the server includes: means for a user to upload data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for the server to verify the consistency and completeness of the uploaded data; means for the server to analyze the text data using natural language processing technology and extract characteristics of the deceased's words and actions; means for the server to analyze the voice data using voice recognition technology and extract the deceased's voice quality and speaking style; means for the server to analyze photos and videos using image recognition technology and extract the deceased's appearance and movements; means for the generation AI to generate a hologram and a voice model based on the generated characteristics of the deceased; means for the server to transfer the hologram and voice model to a hologram device; and means for the user to display a hologram of the deceased and interact with the deceased. This allows the user to interact with the hologram of the deceased and experience a physical sensation of the deceased.

[1321] "User" refers to an individual who accesses the system, uploads data of a deceased person, and uses a hologram device.

[1322] "Server" refers to the device or system that receives, verifies, and analyzes data uploaded by users, and generates and transmits holograms and audio models.

[1323] A "deceased person" refers to a person who has already passed away, but about whom the user wishes to retain memories.

[1324] "Audio data" refers to files containing recordings of the deceased person's voice.

[1325] "Photograph" refers to a still image that visually records the appearance of the deceased.

[1326] "Video" refers to a dynamic image file that includes the movements of the deceased.

[1327] "Social media posts" refers to data such as text, images, and videos posted by the deceased on social media platforms on the Internet.

[1328] "Words and actions" refers to the words and actions of the deceased.

[1329] "Composition" refers to a document written by the deceased.

[1330] A "letter" refers to a document written by a deceased person as a form of personal communication.

[1331] "Data Integrity" refers to the state in which uploaded data is accurate and complete.

[1332] "Natural language processing technology" refers to the technology of analyzing text data and extracting meaning from it.

[1333] "Voice recognition technology" refers to technology that analyzes voice data and extracts features from the voice.

[1334] "Image recognition technology" refers to the technology of analyzing images and videos and extracting features from them.

[1335] "Generative AI" refers to artificial intelligence technology that automatically generates holograms and audio models based on analyzed data.

[1336] "Hologram device" refers to a device that displays generated holograms and allows a user to interact with them.

[1337] "Interaction" refers to a two-way exchange between a user and a hologram.

[1338] The present invention is a system that uses AI to create holograms based on data from the deceased person's life, providing comfort to surviving family members and those involved. Specific embodiments for carrying out the present invention are described below.

[1339] Uploading data

[1340] The user accesses the system and uploads the necessary data about the deceased. This is accomplished by logging into the system's web portal using a home computer or smartphone and uploading photos, audio files, videos, text files, etc. of the deceased. For example, a user can select photos of their deceased grandfather and past messages to be imported into the system.

[1341] Receipt and confirmation of data

[1342] The server receives the uploaded data and checks its consistency and integrity. Specifically, it checks the file format (e.g., jpg, mp3, mp4, txt, etc.) of each file received by the server, and then calculates the file's hash value to ensure the data is not corrupted. If an improper file format or corruption is detected, an error message is returned to the user.

[1343] Data analysis

[1344] The server analyzes the received data and extracts the necessary information. Natural language processing (NLP) technology is used to analyze the text data and extract characteristics of the deceased's words and actions. For example, distinctive phrases and speaking characteristics are extracted from letters and emails uploaded by the user. Speech recognition technology is also used to analyze the audio data and identify the deceased's voice quality and intonation. Image recognition technology is used to analyze the deceased's face and movements from photos and videos. The specific software used at this stage is Python libraries (NLTK, SpaCy, OpenCV, etc.) and machine learning frameworks such as TensorFlow.

[1345] Hologram generation

[1346] The generative AI generates a hologram of the deceased based on the analyzed data. 3D modeling technology is used to create a 3D model that replicates the appearance of the deceased. Voice synthesis technology is used to create an audio file that replicates the deceased's voice. Motion capture technology is then used to create an animation that replicates the natural movements of the deceased. Specific software used at this stage includes Blender (3D modeling) and Vosk (voice synthesis). The prompt is something like, "Generate a 3D hologram of my grandfather based on his old photos and a recorded message."

[1347] Hologram data transfer

[1348] The server transfers the generated hologram data to the hologram device. The generated 3D model, audio file, and motion data are packaged and transferred to the hologram device as a single file. Once the transfer is complete, the user is notified.

[1349] Hologram display and interaction

[1350] The user activates the hologram device, which displays a hologram of the deceased. The user turns on the device and launches a specified application. The hologram device reads the data transferred from the server and displays a 3D hologram of the deceased. It plays audio and runs animations in real time. The user can talk to the hologram, which responds by reflecting the characteristics of the deceased. Enjoying this interaction brings back fond memories of the deceased and brings emotional healing.

[1351] In this way, the invention utilizes data from the deceased person's lifetime to provide a hologram that is extremely valuable to their family and friends.

[1352] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1353] Step 1:

[1354] Users access the system and upload data such as audio, photos, videos, social media posts, words, actions, essays, and letters of the deceased. The input is various data files (e.g., jpg, mp3, mp4, txt, etc.) owned by the user, and the output is the data files imported into the system. Specifically, users log in to the system's web portal using their home computer or smartphone and perform the upload operation.

[1355] Step 2:

[1356] The server receives data uploaded by users and checks its integrity and completeness. The input is the data file uploaded by the user, and the output is the data file whose integrity and completeness have been checked. Specifically, the server verifies the format of each file (e.g., jpg, mp3, mp4, etc.) and calculates a hash value to ensure the file is not corrupted. If the file is corrupted or in an improper format, an error message is returned to the user.

[1357] Step 3:

[1358] The server analyzes the received data and extracts the necessary information. The input is a data file whose consistency and completeness has been confirmed, and the output is a dataset with extracted characteristics of the deceased. Specific operations include using NLP technology to analyze the text data and extract commonly used phrases and speaking characteristics; analyzing the audio data using speech recognition technology to identify the voice quality and intonation of the deceased; and using image recognition technology to extract the face and movements of the deceased from photos and videos. Specific software used is Python libraries (NLTK, SpaCy, OpenCV, etc.) and TensorFlow.

[1359] Step 4:

[1360] The generation AI generates a hologram of the deceased based on the analyzed data. The input is the characteristic data of the deceased analyzed on the server, and the output is a hologram file containing a 3D model, audio file, and motion data. Specific operations involve using 3D modeling technology to create a 3D model that recreates the appearance of the deceased, and using voice synthesis technology to generate an audio file that recreates the deceased's voice. Furthermore, motion capture technology is used to create an animation that recreates the movements of the deceased. The specific software used is Blender (3D modeling) and Vosk (voice synthesis).

[1361] Step 5:

[1362] The server transfers the generated hologram data to the hologram device. The input is the generated hologram file, and the output is the data stored in the hologram device. Specifically, the server packages the generated 3D model, audio file, and motion data and transfers it to the hologram device as a single file. Once the transfer is complete, the server notifies the user that the data transfer is complete.

[1363] Step 6:

[1364] The user starts the hologram device and displays a hologram of the deceased. The input is a hologram file stored in the hologram device, and the output is a hologram of the deceased that is displayed and played back. Specifically, the user turns on the hologram device and launches a specified application. The hologram device reads the data, displays a 3D hologram of the deceased, plays audio in real time, and runs animations.

[1365] Step 7:

[1366] Users can enjoy interacting with the hologram. The input is responses from the hologram device and user input (voice and gestures), and the output is an interactive exchange experience. Specifically, when the user speaks to the hologram, the hologram responds using pre-analyzed data and real-time voice recognition technology. Continuing conversation and interaction with the user can revive fond memories of the deceased, bringing emotional healing.

[1367] (Application example 1)

[1368] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1369] Systems that more realistically recreate memories of the deceased and provide comfort to surviving family and friends face many technical challenges. In particular, systems that not only accurately reflect the characteristics of the deceased but also deepen communication with users through real interactions are required. Another challenge is to personalize the shopping experience and provide special value by recommending products based on the user's purchasing preferences.

[1370] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1371] In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing the text data using natural language processing technology to extract characteristics of the deceased's words and actions; means for analyzing the voice data using voice recognition technology to extract the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology to extract the deceased's appearance and movements; means for generating a hologram and a voice model based on the generated characteristics of the deceased; means for transferring the hologram and the voice model to a hologram device; means for displaying a hologram of the deceased and allowing the user to interact with the deceased; and means for recommending products based on the user's purchasing preferences. This enables the user's purchasing experience to be personalized while realistically recreating the deceased's memories.

[1372] A "deceased person" refers to a person who has passed away and is the person about whom data is to be reproduced.

[1373] "Data upload" refers to a user sending information about a deceased person (such as audio, photos, videos, social media posts, words, actions, writings, letters, etc.) to the system.

[1374] "Data integrity and completeness" means ensuring that uploaded data is accurate, complete, and not corrupted or missing.

[1375] "Natural language processing technology" is a technology that analyzes text data and extracts the characteristics and meaning of sentences and behavior.

[1376] "Voice recognition technology" is a technology that analyzes voice data and extracts characteristics such as speaking style and voice quality.

[1377] "Image recognition technology" is a technology that analyzes photographic and video data to extract a person's appearance and movements.

[1378] A "hologram" is a three-dimensional image reproduced to recreate the appearance and movements of a deceased person.

[1379] A "voice model" is digital voice data generated to reproduce the voice of a deceased person.

[1380] A "hologram device" is a hardware device for displaying generated holograms and audio models.

[1381] "Interaction" means that the user can have a conversation or dynamic interaction with the hologram of the deceased person.

[1382] "Purchasing preferences" refers to purchasing tendencies based on a user's interests and preferences.

[1383] "Product recommendation" refers to the process of presenting relevant products based on a user's purchasing preferences.

[1384] The system for implementing this invention generates a hologram and a voice model using various data of the deceased person, and allows the user to interact with the hologram and voice model and receive personalized product recommendations. Specific embodiments are described below.

[1385] System Overview

[1386] The system consists of the following main components:

[1387] 1. Data upload method

[1388] 2. Means of verifying data consistency and completeness

[1389] 3. Data analysis methods (natural language processing technology, voice recognition technology, image recognition technology)

[1390] 4. Hologram and audio model generation method

[1391] 5. Transfer method to hologram device

[1392] 6. Holographic Display and Interaction Methods

[1393] 7. Product recommendation means

[1394] Hardware and software used

[1395] Hardware:

[1396] Common devices such as smartphones, tablets, and personal computers

[1397] Dedicated hologram device

[1398] software:

[1399] Server Application

[1400] Natural Language Processing (NLP) Engine

[1401] Speech Recognition Engine

[1402] Image Recognition Engine

[1403] Generative AI Models

[1404] Speech synthesis engine

[1405] Product recommendation engine

[1406] Operation flow

[1407] 1. Data upload:

[1408] Users use their own devices (such as smartphones) to upload data such as audio, photos, videos, social media posts, essays, and letters of the deceased to the system.

[1409] 2. Data consistency and integrity checks:

[1410] The server checks the integrity and completeness of the uploaded data, specifically by validating the file format, size, and content, and prompting the user to re-upload if there are any issues.

[1411] 3. Data Analysis:

[1412] The server will analyze the uploaded data using the following analysis techniques:

[1413] Text analysis: Using NLP techniques to extract characteristics of the deceased's words and actions from text data.

[1414] Speech analysis: Using speech recognition technology to extract voice quality and speaking style from audio data.

[1415] Image analysis: Using image recognition technology to analyze the appearance and movements of the deceased from photographs and videos.

[1416] 4. Hologram and audio model generation:

[1417] A generative AI model is used to generate a hologram and voice model of the deceased person based on the acquired feature data.

[1418] 5. Transfer to hologram device:

[1419] The generated hologram and audio model are packaged and transferred to the hologram device.

[1420] 6. Hologram Display and Interaction:

[1421] The user activates the hologram device and interacts with the hologram of the deceased person. The hologram responds to the user's input, realizing a dialogue with the user.

[1422] 7. Product recommendation:

[1423] Using a product recommendation engine, the hologram suggests products based on the user's purchasing preferences, just like a shop assistant, and explains the reasons and features of the recommended products.

[1424] Prompt Sentence Examples

[1425] Examples:

[1426] User: The user opens a shopping app on their smartphone and enjoys shopping while viewing a hologram of their deceased mother. When the user says, "I want a new shirt," the hologram suggests several shirts and explains why it recommends them.

[1427] Example prompt sentence:

[1428] text

[1429] Show me a hologram of a deceased person who knows my preferences and suggests products in a specific category (e.g., "shirts"), how I chose them, and why I recommend them.

[1430] Using this system, users can not only recreate memories of their deceased loved ones, but also have a personalized shopping experience through interactions with the deceased.

[1431] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1432] Step 1:

[1433] A user uses a device (smartphone or tablet) to upload data about the deceased (audio, photos, videos, social media posts, words, actions, essays, letters, etc.) to the system. These files are sent to the server as input data. The server temporarily stores the received data as output.

[1434] Step 2:

[1435] The server verifies the integrity and completeness of the uploaded data. As input, it validates the uploaded data file, checking the file format, size, whether it is corrupted, and ensuring that all data has been received correctly. As output, it generates a flag indicating whether the data is valid.

[1436] Step 3:

[1437] The server analyzes the received data. The input data is the deceased person's data received in step 1. Specifically, the following process is performed:

[1438] Using natural language processing technology, characteristics of the deceased's words and actions are extracted from text data.

[1439] Using voice recognition technology, voice quality and speaking style are extracted from audio data.

[1440] Using image recognition technology, the appearance and movements of the deceased are extracted from photographs and videos.

[1441] As an output, feature data of the deceased extracted by each technique is generated.

[1442] Step 4:

[1443] A generative AI model is used to generate a hologram and voice model based on the analyzed feature data. The feature data obtained in step 3 is used as input. Specifically, the appearance of the deceased is recreated using 3D modeling software, and the voice is recreated using voice synthesis technology. The output is hologram data and voice data.

[1444] Step 5:

[1445] The server transfers the generated hologram and audio data to the hologram device. As input, the data generated in step 4 is used. As output, it is sent to the hologram device, where it is converted into a displayable format.

[1446] Step 6:

[1447] The user activates the hologram device and displays a hologram of the deceased person. As input, hologram and audio data are loaded into the hologram device. Specifically, the user turns on the device and launches a dedicated application. As output, a hologram of the deceased person is displayed and interaction begins.

[1448] Step 7:

[1449] The hologram recommends products based on the user's purchasing preferences. The user's purchasing history and preference data are used as input. Specifically, the product recommendation engine analyzes the user's preferences and generates a product list based on them. As output, the hologram explains to the user the recommended products, their features, and the reasons for the recommendation.

[1450] Prompt Sentence Examples

[1451] Show me a hologram of a deceased person who knows my preferences and suggests products in a specific category (e.g., "shirts"), how I chose them, and why I recommend them.

[1452] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1453] This invention is a system that uses generative AI to create a hologram based on data from the deceased's life, and combines it with an emotion engine to respond according to the user's emotional state. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[1454] Uploading data

[1455] A user accesses the system and uploads data about the deceased. For example, a user accesses the system on a home computer or smartphone and uploads photos of a deceased grandfather or past messages.

[1456] Receipt and confirmation of data

[1457] The server receives the uploaded data and checks its consistency and completeness. For example, the server validates the uploaded file format and size, ensures that images load properly, and that audio files are not corrupted.

[1458] Data analysis

[1459] The server analyzes the received data and extracts the necessary information.

[1460] Using natural language processing (NLP) technology, the characteristics of the deceased's words and actions are analyzed from text data.

[1461] Using voice recognition technology, characteristic voice qualities and speaking styles are analyzed from audio data.

[1462] Using image recognition technology, the server analyzes the face and body language of the deceased from photos and videos. For example, the server can understand the context of messages and letters from the deceased and identify common phrases and speaking characteristics.

[1463] Hologram generation

[1464] The generating AI generates a hologram of the deceased person based on the analyzed data.

[1465] 3D modeling technology is used to create a 3D model that recreates the appearance of the deceased.

[1466] Voice synthesis technology is used to create an audio file that reproduces the voice of the deceased.

[1467] Motion capture technology is used to create animations that replicate the movements of the deceased, while generative AI creates a 3D model of your grandfather based on old photos, and generates natural-sounding conversational voices based on audio recordings.

[1468] Hologram data transfer

[1469] The server transfers the generated hologram data to the hologram device, packages the generated 3D model, audio file, and motion data, and transfers the package to the hologram device.

[1470] User Notifications

[1471] The server notifies the user that the hologram data transfer has been completed. For example, the server notifies the user by sending a message to the user's device via a pop-up screen or email saying "The hologram data has been transferred to your device."

[1472] Hologram display

[1473] The user activates the hologram device and displays the hologram of the deceased person, e.g., by turning on the device, launching an application, and displaying the transferred hologram of the deceased person.

[1474] Interaction and Emotion Recognition

[1475] Users can enjoy interacting with the holograms. By speaking to the holograms or giving them specific instructions, users can interact with the holograms and relive memories of their deceased loved ones.

[1476] Use of emotion engine

[1477] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness."

[1478] The hologram's behavior and responses are tailored based on the identified emotional state: for example, if the user is sad, the hologram will offer a comforting message or a reminiscence.

[1479] The emotion engine has a means to play pre-set memories and messages according to a specific emotional state. For example, if the user is emotional, the hologram will convey a warm message according to that emotion.

[1480] As a result, this system effectively utilizes data from the deceased person's lifetime and provides optimal responses according to the user's emotional state, thereby achieving deep healing.

[1481] The processing flow will be explained below.

[1482] Step 1:

[1483] Users access the system and upload data about the deceased. For example, they log in to a dedicated web application on their home computer or smartphone, select files such as photos, audio, video, social media posts, essays, and letters of the deceased, and press the upload button to send them to the server.

[1484] Step 2:

[1485] The server receives the uploaded data and checks its integrity and completeness. Specifically, it checks the file format and size of the data received, ensures that images are loaded properly, and that audio files are not corrupted. If there are any abnormalities, it sends a notification to the user, encouraging them to re-upload.

[1486] Step 3:

[1487] The server analyzes the received data. Specifically,

[1488] Using natural language processing (NLP) technology, characteristics of the deceased's words and actions are extracted from text data (letters, social media posts, etc.).

[1489] Voice recognition technology is used to extract the voice quality and speaking style of the deceased from audio data (such as recordings).

[1490] Image recognition technology is used to extract facial and movement characteristics of the deceased from photographs and videos.

[1491] Step 4:

[1492] The AI ​​generates a hologram of the deceased person based on the analyzed data.

[1493] Using 3D modeling technology, a 3D model is created that recreates the appearance of the deceased based on collected facial and body features.

[1494] Using voice synthesis technology, an audio file is created that reproduces the voice of the deceased based on the extracted voice quality and speaking characteristics.

[1495] Motion capture technology is used to create animations that recreate the movements of the deceased.

[1496] Step 5:

[1497] The server transfers the generated hologram data to the hologram device. Specifically, it combines the generated 3D model, audio file, and motion data into a single package and sends it to the hologram device.

[1498] Step 6:

[1499] The server notifies the user that the hologram data transfer has been completed. Specifically, the server sends a pop-up notification or email to the user's device stating, "Hologram data has been transferred to your device."

[1500] Step 7:

[1501] The user activates the hologram device and displays the hologram of the deceased. Specifically, the user turns on the hologram device, launches the dedicated application, reads the transferred hologram data, and displays the hologram of the deceased.

[1502] Step 8:

[1503] Users can enjoy interacting with holograms. By talking to the hologram or giving specific instructions, the hologram will respond, establishing a dialogue.

[1504] Step 9:

[1505] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. Specifically, the hologram device captures the user's facial expressions and tone of voice through a built-in camera and microphone, and the emotion engine analyzes the data to determine emotions such as "sadness," "joy," and "surprise."

[1506] Step 10:

[1507] The server adjusts the hologram's behavior and responses based on the identified emotional state. For example, if the user is sad, the hologram will offer comforting words or warm messages, including reminiscences. If the user is happy, the hologram will engage in proactive interactions tailored to that emotion.

[1508] Step 11:

[1509] The server plays pre-set memories and messages according to the emotional state identified by the emotion engine. For example, if the user is emotional, a special message corresponding to that emotion will be played through the hologram.

[1510] Through the above specific processing steps, the system effectively utilizes data from the deceased person's lifetime and provides appropriate responses according to the user's emotional state, thereby achieving emotional healing.

[1511] Example 2

[1512] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1513] In today's world, there is a need for technology that can preserve the memory and presence of the deceased. However, conventional methods have difficulty reproducing the detailed characteristics of the deceased, and they are unable to respond to the user's emotional state. Therefore, there is a need for technology that allows users to feel deep healing through interactions with the deceased.

[1514] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing text data using natural language processing technology to extract characteristics of the deceased's words and actions; means for analyzing voice data using voice recognition technology to extract the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology to extract the deceased's appearance and movements; means for generating a hologram and voice model of the deceased using a generative AI based on the analyzed data; means for transferring the generated hologram and voice model to a hologram device; means for displaying a hologram of the deceased and allowing the user to interact with the deceased; means for analyzing the user's voice and facial expression data and using an emotion engine to identify the user's emotional state; and means for adjusting the hologram's behavior and responses based on the identified emotional state. This enables a high level of reproduction of the deceased's memories and enables appropriate responses tailored to the user's emotional state.

[1515] "Means for uploading data" means the ability for a user to access the system and send data to the system, such as audio, photographs, videos, social media posts, statements, writings, letters, etc., of the deceased.

[1516] "Means for verifying integrity and completeness" refers to the functionality by which the server verifies that data uploaded to the system is in the proper format and is not corrupted.

[1517] "Means of analyzing text data using natural language processing technology and extracting characteristics of speech and behavior" refers to technology for analyzing phrases and speaking styles frequently used by the deceased from uploaded text data.

[1518] "Means for analyzing audio data and extracting voice quality and speaking style" refers to technology that analyzes uploaded audio data and identifies the tone of the deceased's voice and speaking style characteristics.

[1519] "Means of analyzing images and videos to extract appearance and movements" refers to technology for detecting the facial features and physical movements of the deceased from uploaded photos and videos.

[1520] "Means for generating holograms and audio models using generative AI" refers to technology for generating 3D models and audio files of deceased people based on analyzed data.

[1521] The "means for transferring the hologram and audio model to the hologram device" is a function for packaging the generated data and sending it to the device that displays the hologram.

[1522] "Means for displaying and interacting with holograms" refers to a function that allows a user to use a hologram device to display and interact with a hologram of a deceased person.

[1523] "Means using an emotion engine" refers to technology that analyzes the user's voice and facial expressions to identify their current emotional state.

[1524] The "means for adjusting the hologram's actions and responses" is a function for appropriately changing the hologram's actions and speech based on the user's emotional state.

[1525] This invention is a system that uses data from the deceased's life to create a hologram using a generative AI model and combines it with an emotion engine to respond according to the user's emotional state. The system uses data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters.

[1526] Uploading data

[1527] Users access the system and upload data about the deceased. For example, a user accesses the system's website from a home computer or smartphone and uploads photos of their deceased grandfather or past messages. This uploaded data is sent to the server.

[1528] Receipt and confirmation of data

[1529] The server receives the uploaded data and checks its consistency and completeness. The server validates the uploaded file format (JPEG, PNG, etc.) and size (e.g., under 5MB), and ensures that images are loaded properly and audio files are not corrupted. This ensures that the data used by the system is available without any issues.

[1530] Data analysis

[1531] The server analyzes the received data and extracts the necessary information using the following techniques:

[1532] Natural language processing technology (NLP): Analyzes the deceased's frequently used phrases and speaking characteristics from uploaded text data.

[1533] Voice recognition technology: Analyzes voice data to determine the voice quality and speaking style of the deceased.

[1534] Image recognition technology: Analyzes the face and gestures of the deceased from photographs and videos.

[1535] For example, the server can understand the context of messages and letters from the deceased, identify common phrases and speaking characteristics, and even identify vocal characteristics from audio recordings.

[1536] Hologram generation

[1537] The AI ​​then generates a hologram of the deceased person based on the analyzed data, using the following techniques:

[1538] 3D modeling software (e.g. Blender): Create a 3D model of the deceased's appearance.

[1539] Speech synthesis software (e.g., Google Text-to-Speech): Generates an audio file and reproduces the deceased person's voice.

[1540] Motion capture technology (e.g., Kinect): Creating animations that replicate the movements of the deceased.

[1541] For example, generative AI can create a 3D model of your grandfather based on old photos, or generate natural-sounding conversational voices based on audio recordings.

[1542] Hologram data transfer

[1543] The server transfers the generated hologram data to the hologram device, packages the generated 3D model, audio file, and motion data, and sends the package to the hologram device. When the transfer is complete, the server records the completion of the data transfer.

[1544] User Notifications

[1545] The server notifies the user that the hologram data transfer has been completed. For example, the server sends a notification email to the user's terminal stating "Hologram data has been transferred to the device." It also displays a pop-up notification on the system's web page.

[1546] Hologram display

[1547] The user activates the hologram device and displays the hologram of the deceased. For example, the user turns on the hologram device, launches the application, and clicks the "Show hologram of the deceased" button, which displays the transferred hologram of the deceased.

[1548] Interaction and Emotion Recognition

[1549] Users can enjoy interacting with the hologram. By talking to the hologram or giving specific instructions, users can interact with the hologram and recreate memories of their deceased loved one. The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness."

[1550] Use of emotion engine

[1551] The hologram's behavior and responses are adjusted based on the identified emotional state. For example, if the user is sad, the hologram will provide a comforting message or a reminiscence. The emotion engine has the means to play pre-set memories and messages in response to specific emotional states. For example, if the user is moved, the hologram will deliver a warm message that corresponds to that emotion.

[1552] Prompt Sentence Examples

[1553] "Generate a hologram of the deceased person based on the following data: photographs, voice recordings, messages, letters, and design it to respond appropriately based on the user's emotional state."

[1554] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1555] Step 1:

[1556] The user accesses the system and uploads the deceased's data. The user accesses the system's website using a home computer or smartphone and clicks the "Upload" button. Next, on the upload screen, the user selects the deceased's audio, photos, videos, social media posts, words and actions, essays, letters, etc., and presses the "Send" button to send the data to the server. The input is the deceased's data file, and the output is the data uploaded to the server.

[1557] Step 2:

[1558] The server receives the uploaded data and checks its consistency and completeness. The server validates the received file format (JPEG, PNG, MP3, etc.) and size (e.g., less than 5MB), and ensures that images are loaded in the correct format and audio files are not corrupted. For example, it checks the header of a JPEG file to ensure that the image is valid. The input is the uploaded data file, and the output is the validated data file.

[1559] Step 3:

[1560] The server analyzes the received data and extracts the necessary information. Specifically, it uses natural language processing technology to analyze the text data and extract characteristics of the deceased's words and actions. It also uses speech recognition technology to analyze the audio data and extract the deceased's voice quality and speaking style, and image recognition technology to extract the deceased's appearance and movements from photos and videos. For example, it extracts frequently used phrases from the deceased's text messages. The input is verified data, and the output is analyzed data (text features, audio features, and image features).

[1561] Step 4:

[1562] The generative AI generates a hologram and voice model of the deceased based on the analyzed data. It creates a 3D model of the deceased using 3D modeling software (e.g., Blender), generates an audio file using speech synthesis software (e.g., Google Text-to-Speech), and creates animation data using motion capture technology (e.g., Kinect). The input is the analyzed data, and the output is the generated 3D model and audio file.

[1563] Step 5:

[1564] The server transfers the generated hologram data to the hologram device. The server then combines the generated 3D model, audio file, and motion data into a single package and transfers the package to the hologram device. For example, the server sends the data using the FTP protocol. The input is the generated hologram data, and the output is the data transferred to the hologram device.

[1565] Step 6:

[1566] The server notifies the user that the hologram data transfer is complete. The server sends a notification email to the user's email address stating "Hologram data has been transferred to the device" and also displays a pop-up notification on the system's web page. The input is the transfer completion status, and the output is the notification email and pop-up notification to the user.

[1567] Step 7:

[1568] The user turns on the hologram device and displays a hologram of the deceased. The user turns on the hologram device, launches the application, and clicks the "Show hologram of the deceased" button. The input is the hologram device, and the output is the displayed hologram of the deceased.

[1569] Step 8:

[1570] Users can enjoy interacting with holograms. By speaking to the hologram or giving specific instructions, users can interact with the hologram and recreate memories of their deceased loved ones. The input is the user's voice and actions, and the output is the hologram's response.

[1571] Step 9:

[1572] The emotion engine analyzes the user's voice and facial expression data to identify their emotional state. For example, if the user shows a sad expression or tone of voice, the emotion engine analyzes it and identifies the emotion of "sadness." The input is the user's voice and facial expression data, and the output is the identified emotional state.

[1573] Step 10:

[1574] The hologram's behavior and response are adjusted based on the identified emotional state. For example, if the user is sad, the hologram may offer comforting messages such as, "Are you OK? Is something wrong?" The input is the identified emotional state, and the output is the adjusted hologram's behavior and response.

[1575] (Application example 2)

[1576] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1577] In a hologram system based on data from the deceased's lifetime, when a user enjoys interacting with a hologram of the deceased, it is necessary to obtain appropriate responses according to the user's emotional state. In particular, when displaying holograms using smart glasses in a physical store, it is necessary to realize natural responses and interactions according to changes in the user's emotions.

[1578] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for uploading data such as the deceased's voice, photos, videos, social media posts, words and actions, essays, and letters; means for verifying the consistency and completeness of the uploaded data; means for analyzing text data using natural language processing technology and extracting characteristics of the deceased's words and actions; means for analyzing voice data using voice recognition technology and extracting the deceased's voice quality and speaking style; means for analyzing photos and videos using image recognition technology and extracting the deceased's appearance and movements; means for generating a hologram and voice model based on the deceased's data before death and the user's emotional state; means for transferring the generated hologram and voice model to a hologram device; means for the user to enjoy interaction with the deceased in a physical store using smart glasses; means for analyzing the user's voice and face data using an emotion engine and identifying the user's emotional state; and means for adjusting the hologram's behavior and response based on the identified emotional state. This makes it possible to achieve optimal responses and natural interactions according to the user's emotional state.

[1579] "Voice of the deceased" refers to voice data recorded by the deceased during their lifetime.

[1580] A "photograph" is a still image of the deceased or related objects or people.

[1581] "Video" is dynamic image data of the deceased or related objects or people.

[1582] "Social media posts" are content such as text, images, and videos posted by the deceased on social media while they were alive.

[1583] "Words and actions" is a record of the words and actions of the deceased during their lifetime.

[1584] "Compositions" refer to essays and writings written by the deceased during their lifetime.

[1585] A "letter" is a letter or message written by a deceased person while they were still alive.

[1586] "Means for uploading" refers to the procedures and interfaces that allow users to send data about the deceased to the server.

[1587] "Integrity and completeness check measures" are processes that verify that uploaded data is in the correct format and is complete.

[1588] "Natural language processing technology" is a technology for analyzing text data and extracting meaning and characteristics.

[1589] "Voice recognition technology" is a technology that analyzes voice data to identify the speaker's voice quality and speaking style.

[1590] "Image recognition technology" is a technology that analyzes images and videos to extract the appearance and movements of a target.

[1591] An "emotion engine" is a technology that identifies a user's emotional state from their voice and facial expressions and adjusts responses accordingly.

[1592] A "hologram device" is a device for displaying a generated hologram.

[1593] "Smart glasses" are augmented reality (AR) display devices worn by users.

[1594] "Means for enjoying interaction" are functions and processes that allow users to interact with holograms through dialogue and manipulation.

[1595] The system for carrying out the present invention includes the following processing flow and data analysis process.

[1596] 1. Upload your data

[1597] Users access the system using their home computers or smartphones and upload data such as audio, photos, videos, social media posts, words and actions, essays, and letters of the deceased, thereby collecting information about the deceased's activities and characteristics while they were alive.

[1598] 2. Receipt and confirmation of data

[1599] The server receives the data uploaded by the user and checks the consistency and completeness of each data item. This includes validating the file format and size, and testing the readability of image and audio files. For example, the server checks whether the received image file is corrupted.

[1600] 3. Data Analysis

[1601] The server analyzes the received data and extracts the required information using the following techniques:

[1602] Natural language processing technology (NLP): Analyzes the characteristics of the deceased's words and actions from text data.

[1603] Voice recognition technology: Analyzes the voice quality and speaking style of the deceased from audio data.

[1604] Image recognition technology: Analyzes the face and gestures of the deceased from photographs and videos.

[1605] 4. Hologram Generation

[1606] Based on the deceased's life data and the user's emotional state, the generative AI generates a hologram and voice model. Specifically, it uses 3D modeling technology to recreate the deceased's appearance, voice synthesis technology to generate the deceased's voice, and motion capture technology to recreate their movements and behavior.

[1607] 5. Transfer of hologram data

[1608] The server transfers the generated hologram data to the hologram device, which includes a package of 3D models, audio files, and motion data. After the transfer is complete, the server notifies the user of the transfer to their smart glasses.

[1609] 6. Interaction and Emotion Recognition

[1610] Users interact with holograms in brick-and-mortar stores using smart glasses. The smart glasses' cameras and microphones capture the user's voice and facial expressions and transmit them to a server. The server uses an emotion engine to analyze the user's emotional state and adjust the hologram's responses and behavior based on the identified emotion. For example, if the user has a sad expression, the hologram will convey a comforting message such as "Cheer up."

[1611] Examples of prompt statements

[1612] Prompt statement:

[1613] Recognize the user's emotional state and generate a hologram that responds based on that state. Take the following data as input:

[1614] 1. User's voice data: voice_sample.wav

[1615] 2. User's facial expression image: face_sample.png

[1616] 3. Text data of the deceased:grandfather_messages.txt

[1617] The generated hologram will have animation and audio that will convey an uplifting message if the user is feeling sad.

[1618] This allows users to recreate memories of the deceased in a physical store and receive appropriate responses based on their emotions.

[1619] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1620] Step 1:

[1621] Users access the system using their home computers or smartphones and upload data about the deceased, such as audio, photos, videos, social media posts, words and actions, essays, letters, etc. The input is various data related to the deceased, and the output is this data stored on the server.

[1622] Step 2:

[1623] The server receives the uploaded data and checks the consistency and completeness of each piece of data. The input is the file uploaded by the user, and the output is the verified data. Specifically, it validates the file format and size, and performs read tests on image and audio files. For example, the server checks whether the received image file is corrupted.

[1624] Step 3:

[1625] The server uses natural language processing technology to analyze text data and extract characteristics of the deceased's words and actions. The input is text data, and the output is data that represents the characteristics of the deceased's words and actions. Specifically, it uses an NLP library to perform context analysis and key phrase extraction.

[1626] Step 4:

[1627] The server uses voice recognition technology to analyze the voice data and extract the voice quality and speaking style of the deceased. The input is the voice data, and the output is characteristic data of the voice quality and speaking style of the deceased. Specifically, it uses a voice recognition library to analyze pitch and tone from the voice waveform.

[1628] Step 5:

[1629] The server uses image recognition technology to analyze photos and videos to extract the appearance and movements of the deceased. The input is photo and video data, and the output is data for recreating the appearance and movements of the deceased. Specific operations include facial recognition and motion capture using an image processing library.

[1630] Step 6:

[1631] The server uses generative AI to generate a hologram and voice model based on the deceased's data and the user's emotional state. The input is the analyzed characteristic data of the deceased and the user's emotional data, and the output is the hologram and voice model data. Specifically, the hologram is created using 3D modeling, voice synthesis, and motion capture technology.

[1632] Step 7:

[1633] The server packages the generated hologram data and transfers it to the hologram device. The input is the generated hologram and audio model data, and the output is data in a format that can be displayed on the hologram device. Specifically, data compression and transfer protocols are used to ensure rapid data transfer.

[1634] Step 8:

[1635] The user enjoys interacting with the deceased person in a physical store using smart glasses. The input is data transmitted to the hologram device, and the output is a displayed hologram. Specifically, the user launches the smart glasses application and begins interacting with the displayed hologram.

[1636] Step 9:

[1637] The user's voice and facial expression data are captured by the smart glasses and sent to the server. The input is the user's voice and facial expression data, and the output is trigger data to start the analysis. Specifically, the data is captured using the microphone and camera of the smart glasses and sent to the server via the network.

[1638] Step 10:

[1639] The server uses an emotion engine to analyze the user's voice and facial expression data to identify the user's emotional state. The input is the user's voice and facial expression data, and the output is the user's emotional state data. Specifically, the server applies voice analysis and facial expression analysis algorithms to analyze emotions.

[1640] Step 11:

[1641] The server adjusts the hologram's behavior and response based on the identified emotional state. The input is the user's emotional state data, and the output is the adjusted hologram response data. Specifically, it uses an AI model to generate and display emotionally appropriate hologram responses.

[1642] Examples of prompts:

[1643] Prompt statement:

[1644] Recognize the user's emotional state and generate a hologram that responds based on that state. Take the following data as input:

[1645] 1. User's voice data: voice_sample.wav

[1646] 2. User's facial expression image: face_sample.png

[1647] 3. Text data of the deceased:grandfather_messages.txt

[1648] The generated hologram will have animation and audio that will convey an uplifting message if the user is feeling sad.

[1649] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1650] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1651] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1652] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1653] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1654] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1655] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1656] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1657] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1658] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1659] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1660] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1661] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1662] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1663] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1664] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1665] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1666] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1667] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1668] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1669] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1670] The following is further disclosed regarding the above embodiment.

[1671] (Claim 1)

[1672] A means to upload data such as audio, photos, videos, social media posts, words, actions, writings, letters, etc. of the deceased;

[1673] a means of verifying the integrity and completeness of the uploaded data;

[1674] A means for analyzing text data using natural language processing technology and extracting characteristics of the deceased's words and actions;

[1675] A means for analyzing the audio data using voice recognition technology to extract the voice quality and speaking style of the deceased;

[1676] A method of analyzing photos and videos using image recognition technology to extract the appearance and movements of the deceased, and

[1677] a means for generating a hologram and a voice model based on the generated characteristics of the deceased;

[1678] means for transferring the hologram and the audio model to a hologram device;

[1679] a means for displaying a hologram of the deceased person and for a user to interact with the deceased person;

[1680] A system including:

[1681] (Claim 2)

[1682] 10. The system of claim 1, further comprising means for notifying a user of the generated hologram and audio model.

[1683] (Claim 3)

[1684] 10. The system of claim 1, further comprising a notification means for prompting a user to re-upload if the uploaded data is corrupted.

[1685] "Example 1"

[1686] (Claim 1)

[1687] A means for users to upload data such as audio, photos, videos, social media posts, statements, writings, letters, etc. of the deceased;

[1688] a means by which the server verifies the integrity and completeness of the uploaded data;

[1689] A means for the server to analyze the text data using natural language processing technology and extract characteristics of the deceased's words and actions;

[1690] A means for the server to analyze the voice data using voice recognition technology and extract the voice quality and speaking style of the deceased;

[1691] The server uses image recognition technology to analyze photos and videos to extract the appearance and movements of the deceased,

[1692] A means for generating a hologram and a voice model based on the characteristics of the deceased person generated by the generation AI; ...

Claims

1. A means to upload data such as audio, photos, videos, social media posts, words, actions, writings, letters, etc. of the deceased; a means of verifying the integrity and completeness of the uploaded data; A means for analyzing text data using natural language processing technology and extracting characteristics of the deceased's words and actions; A means for analyzing the audio data using voice recognition technology to extract the voice quality and speaking style of the deceased; A method of analyzing photos and videos using image recognition technology to extract the appearance and movements of the deceased, and a means for generating a hologram and a voice model based on the generated characteristics of the deceased; means for transferring the hologram and the audio model to a hologram device; a means for displaying a hologram of the deceased person and for a user to interact with the deceased person; A system including:

2. The system of claim 1 further comprising means for notifying a user of the generated hologram and sound model.

3. 2. The system of claim 1, further comprising a notification means for prompting a user to re-upload if the uploaded data is corrupted.

Images