System

Abstract

To provide a system allowing general consumers to experience conversations with celebrities or the deceased.SOLUTION: A system for allowing general consumers to fulfill desire to have conversations with celebrities or the deceased includes: collection means for collecting public information of celebrities or the deceased and personal data including content of past postings on social networking services; means for generating a prompt sentence for instructing generation of an answer of a celebrity or the deceased to a question from a user; means for using the generated prompt sentence, the personal data of the celebrity or the deceased collected by the collection unit and a generative AI model to generate a celebrity's or deceased's answer to the question from the user; and voice synthesis means for using voice synthesis software to vocalize the generated answer as conversation that mimics characteristics of the celebrity or the deceased.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] While there is a demand from ordinary consumers wanting to converse with celebrities and deceased people, there is a problem in that there is no way to make this happen. [Means for solving the problem]

[0005] As a means of solving this problem, we provide a system that combines a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, and an image recognition / synthesis means. Specifically, the generative AI generates conversation that imitates the characteristics of celebrities or deceased people based on the collected personal data, the voice recognition / synthesis means converts the generated conversation into audio, and the image recognition / synthesis means generates images of the celebrities or deceased people. This allows general consumers to experience conversations with celebrities and deceased people. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] As one embodiment of the present invention, a system is provided that allows general consumers to experience conversations with celebrities and deceased persons. This system is configured by combining a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, and an image recognition / synthesis means. The personal data collection means collects personal data such as public information of celebrities and deceased persons and SNS posts. The generative AI generates a conversation that imitates the characteristics of the celebrity or deceased person based on the collected personal data. The voice recognition / synthesis means converts the generated conversation into voice. The image recognition / synthesis means generates an image of the celebrity or deceased person. By combining these means, a system is realized that allows general consumers to experience conversations with celebrities and deceased persons.

[0029] "Example 2"

[0030] As a specific example, consider a system that allows general consumers to experience a conversation with their deceased grandparents. First, a personal data collection means collects personal data such as the grandparents' past letters and diaries, and testimonies from family members. A generative AI generates conversation that mimics the characteristics of the grandparents based on the collected personal data. A voice recognition / synthesis means synthesizes the generated conversation into a voice that is close to the grandparents' voice. An image recognition / synthesis means generates images of the grandparents using 3D models generated from photos of the grandparents. By combining these means, a system is realized that allows general consumers to experience a conversation with their deceased grandparents.

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: Personal data such as publicly available information and social media posts about celebrities and deceased people is collected using personal data collection methods.

[0034] Step 2: Generative AI generates conversations that mimic the characteristics of celebrities or deceased people based on the collected personal data.

[0035] Step 3: The speech recognition / synthesis means converts the generated speech into audio.

[0036] Step 4: Image recognition / synthesis procedures generate an image of the celebrity or deceased person.

[0037] "Example 2"

[0038] Step 1: Personal data collection instruments are used to collect personal data such as grandparents' past letters and diaries, and testimonies from family members.

[0039] Step 2: The generative AI generates conversations that mimic the characteristics of the grandparents based on the collected personal data.

[0040] Step 3: The speech recognition / synthesis means synthesizes the generated conversation into a voice that resembles the grandparents' voice.

[0041] Step 4: The image recognition / synthesis means generates images of the grandparents using the 3D models generated from the photos of the grandparents.

[0042] Example 1

[0043] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0044] There is a demand for a more realistic and natural conversation experience in a system that allows ordinary consumers to fulfill their wish to talk to celebrities or deceased people. However, conventional systems have limitations in the quality of collected data, the naturalness of generated conversations, and the realism of voice and images, resulting in an insufficient user experience.

[0045] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0046] In this invention, the server includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition means, a voice synthesis means, an image recognition means, and an image generation means, which enable the server to generate a conversation that imitates the characteristics of a specific person based on the collected personal data, convert a user's voice input into text, vocalize the generated conversation, and recognize and generate an image of the specific person.

[0047] "Personal data collection means" refers to means for collecting data such as publicly available information about a specific person and the contents of social media posts.

[0048] "Generative AI" is AI that generates conversations that mimic the characteristics of a specific person based on collected data.

[0049] A "speech recognition means" is a means for converting a user's voice input into text.

[0050] The "voice synthesis means" is a means for converting the generated conversation into voice.

[0051] The "image recognition means" is a means for recognizing an image of a specific person.

[0052] An "image generating means" is a means for generating an image of a particular person.

[0053] This invention is a system for fulfilling the desire of ordinary consumers to converse with celebrities or deceased people, and includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition means, a voice synthesis means, an image recognition means, and an image generation means.

[0054] Personal data collection methods

[0055] The server collects personal data such as public information and social media posts of celebrities and deceased people. This data is collected using web scraping tools (e.g., Beautiful Soup and Scrapy) and social media APIs (e.g., Twitter API and Facebook Graph API). The collected data is stored in a database.

[0056] Conversation Generation Method

[0057] The server generates a conversation using generative artificial intelligence (e.g., GPT-4 (registered trademark) by OpenAI (registered trademark)) based on the collected personal data. The generative artificial intelligence analyzes the input personal data and generates natural conversational sentences. An example of a prompt sentence is, "Based on the public information and SNS posts of famous singer A, please generate a conversation that imitates his characteristics."

[0058] Speech recognition and synthesis means

[0059] The server converts the generated conversational text into voice using speech synthesis software (e.g., Google® Text-to-Speech or Amazon Polly). It also converts the user's voice input into text using speech recognition software (e.g., Google Speech-to-Text or IBM Watson® Speech to Text). This allows the server to obtain the user's question as text and input it into the generative artificial intelligence.

[0060] Image recognition and image generation

[0061] The server uses image generation software (e.g. DALL-E or DeepArt) to generate images of celebrities or deceased people, allowing the user to visually recognize the person they are speaking with. It inputs prompts to the image generation software based on the collected personal data, and sends the generated image to the user's device.

[0062] User interaction

[0063] Users access the system through a terminal (e.g., a smartphone or PC) and experience conversations with celebrities or deceased people. When the user asks a question by voice, the voice recognition means converts the question into text, and the generative artificial intelligence generates an answer. The generated answer is converted into voice by the voice synthesis means and returned to the user. At the same time, an image of the celebrity or deceased person, generated by the image recognition means and the image generation means, is displayed.

[0064] Examples

[0065] For example, if a user wishes to "talk to famous singer A," the system operates as follows.

[0066] 1. The server collects public information and social media posts about famous singer A.

[0067] 2. The server uses generative artificial intelligence based on the collected data to generate conversations that mimic the characteristics of famous singer A.

[0068] 3. The server converts the generated conversational text into voice using speech synthesis software.

[0069] 4. The server generates an image of famous singer A.

[0070] 5. The user experiences a conversation with famous singer A through the terminal using audio and images.

[0071] In this way, a system is realized that allows ordinary consumers to experience conversations with celebrities and deceased people.

[0072] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0073] Step 1:

[0074] The server uses personal data collection means to collect public information and social media posts about celebrities and deceased people. Specifically, it uses web scraping tools (e.g. Beautiful Soup and Scrapy) and social media APIs (e.g. Twitter API and Facebook Graph API). The input is the name and related information of a specific person, and the output is the collected personal data. The collected data is stored in a database.

[0075] Step 2:

[0076] The server retrieves the collected personal data from the database and generates a conversation using generative artificial intelligence (e.g., OpenAI's GPT-4). The input is the collected personal data, and the output is the generated conversation. Specifically, a prompt sentence is input into the generative artificial intelligence, and a conversation sentence that imitates the characteristics of a specific person is generated. An example of a prompt sentence is, "Based on the public information of famous singer A and his social media posts, please generate a conversation that imitates his characteristics."

[0077] Step 3:

[0078] The server converts the generated conversation text into voice using speech synthesis software (e.g., Google Text-to-Speech or Amazon Polly). The input is the generated conversation text, and the output is an audio file. Specifically, the generated conversation text is input into the speech synthesis software to generate an audio file.

[0079] Step 4:

[0080] The server converts the user's voice input into text using voice recognition software (e.g., Google Speech-to-Text or IBM Watson Speech to Text). The input is the user's voice, and the output is text. Specifically, the user's voice input is input into the voice recognition software and converted into text.

[0081] Step 5:

[0082] The server generates an image of a specific person using image generation software (e.g., DALL-E or DeepArt) based on the collected personal data. The input is the collected personal data, and the output is the generated image. Specifically, a prompt is input to the image generation software, and an image of a specific person is generated.

[0083] Step 6:

[0084] Users access the system through a terminal (e.g., a smartphone or PC) and experience a conversation with a celebrity or a deceased person. The input is the user's voice question, and the output is the generated audio and image of the conversation. Specifically, when the user asks a question by voice, the voice recognition means converts the question into text, and the generative artificial intelligence generates an answer. The generated answer is converted into voice by the voice synthesis means and returned to the user. At the same time, an image of the celebrity or deceased person, generated by the image recognition means and the image generation means, is displayed.

[0085] (Application example 1)

[0086] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] When ordinary consumers wish to converse with celebrities or deceased people, it is difficult to provide a realistic experience that would allow them to realize their wish. In particular, advanced technology is required to provide a real-time conversation experience that combines voice and images. There is also a demand to provide this experience in a form that is easy for users to use.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0089] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, an application means installed on a smartphone, a means for recognizing a user's voice input and generating a voice response in real time, and a means for generating an image of a celebrity or a deceased person and displaying it during a conversation. This allows general consumers to easily enjoy a real conversation experience with a celebrity or a deceased person.

[0090] "Personal data collection means" refers to means for collecting personal data, such as publicly available information about celebrities or the deceased, or the content of social media posts.

[0091] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of celebrities or deceased people based on collected personal data.

[0092] "Speech recognition / synthesis means" means for recognizing a user's voice input and converting the generated speech into voice.

[0093] "Image recognition / synthesis means" refers to means for generating images of celebrities or deceased people and displaying them during a conversation.

[0094] The "application means installed on a smartphone" refers to a means that is installed on a smartphone and provides an application that enables a user to have a conversation experience with a celebrity or a deceased person.

[0095] The "means for recognizing a user's voice input and generating a voice response in real time" refers to a means for recognizing a user's voice input in real time and generating a voice response in response thereto.

[0096] "Means for generating an image of a celebrity or a deceased person and displaying it during a conversation" refers to means for generating an image of a celebrity or a deceased person and displaying it to a user during a conversation.

[0097] The system for carrying out the present invention is configured as follows. First, the server uses a personal data collection means to collect personal data such as public information and social media posts of celebrities and deceased people. This collected data is input to a conversation generation means using generative AI, and conversations that imitate the characteristics of celebrities and deceased people are generated.

[0098] Next, an application installed on the user's device (smartphone) uses speech recognition / synthesis means to recognize the user's voice input and generates a real-time voice response to convert the generated conversation into voice. It also uses image recognition / synthesis means to generate images of celebrities and deceased people and display them to the user during the conversation.

[0099] Specifically, the server uses the following software and hardware: It uses the OpenAI API for generative AI, the SpeechRecognition library for voice recognition, and the gTTS library for voice synthesis. It uses the OpenCV library for image processing. These software run on the server and work in conjunction with applications installed on the user's smartphone.

[0100] For example, if a user wants to talk to a "historical figure," the server collects personal data about that person and uses generative AI to generate a conversation that mimics the characteristics of that person. When the user asks a question to the smartphone, the voice recognition / synthesis means recognizes the question and generates a voice response in real time. At the same time, the image recognition / synthesis means generates an image of the person and displays it on the smartphone screen.

[0101] An example of a prompt is as follows:

[0102] "Collect personal data about a historical figure and generate answers to the following user questions that mimic the characteristics of that person.

[0103] User: What is your dream?

[0104] AI:

[0105] In this way, users can easily enjoy the experience of having a real conversation with a celebrity or a deceased person.

[0106] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0107] Step 1:

[0108] The server uses the personal data collection means to collect personal data such as public information and social media posts of celebrities and deceased individuals. The input is the name of the celebrity or deceased individual, and the output is the collected personal data. This data collection is done through public information on the Internet and social media APIs.

[0109] Step 2:

[0110] The server inputs the collected personal data into a generative AI to generate a conversation that imitates the characteristics of celebrities and deceased people. The input is the collected personal data, and the output is the generated conversation text. The generative AI uses OpenAI's API to generate a conversation based on the prompt sentence.

[0111] Step 3:

[0112] The user's device (smartphone) recognizes the user's voice input using a speech recognition / synthesis means. The input is the user's voice and the output is the user's question in text form. The SpeechRecognition library is used for speech recognition.

[0113] Step 4:

[0114] The server inputs the user's question text into the generative AI, which generates a voice response in real time. The input is the user's question text, and the output is the generated response text. The generative AI again uses OpenAI's API to generate a response based on the prompt text.

[0115] Step 5:

[0116] The user's terminal inputs the generated response text to a speech synthesis means and converts it into speech. The input is the generated response text, and the output is an audio file. The gTTS library is used for speech synthesis.

[0117] Step 6:

[0118] The server generates images of celebrities and deceased people using image recognition / synthesis methods. The input is the name of the celebrity or deceased person, and the output is the generated image file. The OpenCV library is used for image generation.

[0119] Step 7:

[0120] The user's terminal displays the generated audio and image files. The input is the audio and image files, and the output is visual and auditory feedback to the user. The audio files are played through the speaker, and the image files are shown on the display.

[0121] Example 2

[0122] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] Conventional technology has made it difficult for ordinary consumers to experience conversations with deceased people or specific individuals. In particular, generating natural conversations that reflect the characteristics of the deceased, and realistic voices and images, is technically complex and difficult to achieve. For this reason, there is a demand for a system that can reproduce conversations with the deceased.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0125] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a means for providing a conversation experience by combining the generated voice and image, thereby enabling general consumers to experience a natural conversation with a deceased person or a specific person.

[0126] "Personal data collection means" refers to devices or methods for collecting and digitizing personal information of a specific individual, such as past letters, diaries, and testimonies.

[0127] A "conversation generation method using generative AI" is an artificial intelligence technology for generating conversations that mimic the characteristics of a specific individual based on collected personal data.

[0128] "Speech recognition / synthesis means" refers to technology for converting the generated conversation into voice data and generating a voice that resembles the voice of a specific person.

[0129] "Image recognition / synthesis means" is a technology for generating 3D models and realistic images based on photographs or videos of specific people.

[0130] "Means for combining generated audio and images to provide a conversational experience" refers to an apparatus or method for synchronizing generated audio data and image data to provide a user with an interactive conversational experience.

[0131] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0132] This invention is a system for enabling general consumers to experience conversations with deceased people or specific individuals. The system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a means for providing a conversation experience by combining the generated voice and image.

[0133] Collection of Personal Data

[0134] Users collect personal data such as past letters from grandparents or specific individuals, diaries, and testimonials from family members. These data are digitized using a scanner, voice recording device, and text input software. For example, a user can use a scanner to scan letters from grandparents and save them as text data.

[0135] Conversation generation using generative AI

[0136] The server receives the collected personal data and uses a generative AI model (e.g., a general natural language processing model) to generate a conversation that mimics the characteristics of grandparents or a specific person. Specifically, the server analyzes the data and inputs a prompt sentence into the generative AI model. For example, the server inputs a prompt sentence such as, "Based on the contents of the letter from my grandparents, please generate a conversation that reflects the characteristics of my grandparents."

[0137] Speech Recognition / Synthesis

[0138] The server uses a voice recognition / synthesis means (for example, a general voice synthesis API) to synthesize the generated conversation into a voice that resembles the voice of a grandparent or a specific person. Specifically, the server sends the generated text data to a voice synthesis API and obtains a voice file. For example, the generated conversation, "It's a very nice day today," is synthesized into the voice of a grandparent.

[0139] Image Recognition / Synthesis

[0140] The server uses image recognition / synthesis means (e.g., a general image generation algorithm) to generate a 3D model using photos of the grandparents or a specific person. Specifically, the server analyzes the photo data and inputs it into a 3D model generation algorithm. For example, a 3D model is generated based on a photo of the grandparents' faces, and an avatar that moves in real time is created.

[0141] Providing a conversational experience

[0142] The device combines the generated voice and image to provide the user with a conversation experience with a grandparent or a specific person. Specifically, the device plays the audio file and the 3D model in sync, allowing the user to have an interactive conversation. For example, the user can talk to an avatar of a grandparent while looking at the screen of the device.

[0143] Examples and prompts

[0144] Examples

[0145] A user scans a letter from their grandparents and uploads it to the system. The server analyzes the contents of the letter and uses a generative AI model to generate a conversation that mimics the characteristics of the grandparents. The generated conversation is then converted into voice using a common voice synthesis API, and a 3D model of the grandparent is generated using a common image generation algorithm. Finally, the user can talk to the grandparents through the device.

[0146] Examples of prompt statements

[0147] "Based on the content of the letter below, please generate a conversation that reflects the characteristics of the grandparents. The content of the letter is as follows: 'Dear Grandson, It's a very nice day today and the flowers in the garden are in full bloom. If you were here, we could go for a walk together.'"

[0148] By inputting this prompt into the generative AI model, natural conversation that reflects the characteristics of the grandparents is generated.

[0149] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0150] Step 1:

[0151] The user collects personal data such as past letters, diaries, and testimonials from family members of grandparents or other specific individuals. These data are digitized using a scanner, a voice recording device, and text input software. Specifically, the user uses a scanner to scan letters from grandparents and saves them as text data. The input is physical personal data such as letters and diaries, and the output is digitized text data.

[0152] Step 2:

[0153] The server receives digitized personal data uploaded by the user. It then uses a generative AI model (e.g., a general natural language processing model) to generate a conversation that mimics the characteristics of a specific person. Specifically, the server analyzes the data and inputs a prompt sentence into the generative AI model. For example, the server inputs the prompt sentence, "Based on the contents of my grandparents' letters, please generate a conversation that reflects my grandparents' characteristics." The input is the digitized personal data, and the output is the generated conversation text.

[0154] Step 3:

[0155] The server synthesizes the generated conversation text into a voice that resembles the voice of a specific person using a voice recognition / synthesis means (for example, a general voice synthesis API). Specifically, the server sends the generated text data to a voice synthesis API and obtains an audio file. For example, the generated conversation "It's a very nice day today" is synthesized in the voice of a grandparent. The input is the generated conversation text, and the output is an audio file.

[0156] Step 4:

[0157] The server uses image recognition / synthesis means (e.g., a general image generation algorithm) to generate a 3D model using a photo of a specific person. Specifically, the server analyzes the photo data and inputs it into a 3D model generation algorithm. For example, a 3D model is generated based on a photo of a grandparent's face, and an avatar that moves in real time is created. The input is photo data of a specific person, and the output is a 3D model.

[0158] Step 5:

[0159] The terminal combines the generated audio file with the 3D model to provide the user with a conversation experience with a specific person. Specifically, the terminal plays the audio file and the 3D model in sync, allowing the user to have an interactive conversation. For example, a user can interact with an avatar of their grandparents while looking at the screen of the terminal. The input is the audio file and the 3D model, and the output is an interactive conversation experience.

[0160] (Application example 2)

[0161] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] There are existing systems that allow ordinary consumers to fulfill their desire to converse with deceased relatives or celebrities, but these systems have the problem of lacking realism and limiting the user experience. In addition, there is a lack of means to realize dialogue in a virtual space, making it difficult to provide a more realistic experience. This has led to a demand for a system that allows users to experience a more realistic conversation with deceased relatives or celebrities.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0164] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a display means for realizing a conversation in a virtual space, which allows general consumers to experience a more realistic conversation with a deceased person or a celebrity.

[0165] "Personal data collection methods" are methods for collecting data about individuals, such as past letters, diaries, and testimonies from family members of deceased or famous people.

[0166] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of deceased people or celebrities based on collected personal data.

[0167] The "voice recognition / synthesis means" refers to a means for converting the generated conversation into voice and synthesizing a voice that is close to the voice of the deceased or famous person.

[0168] "Image recognition / synthesis means" refers to a means for generating 3D models from photographs or videos of deceased persons or celebrities and synthesizing realistic images.

[0169] "Display means for realizing dialogue in a virtual space" refers to a means for displaying generated conversations and images in a virtual space, providing a user with the experience of interacting with a deceased person or a celebrity.

[0170] The following system configuration will be described as an embodiment of the present invention.

[0171] System Configuration

[0172] This system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a display means for realizing dialogue in a virtual space.

[0173] Personal data collection methods

[0174] The server collects data about deceased or famous people, such as past letters, diaries, and testimonials from family members. This data may be provided by users or collected from public sources on the Internet.

[0175] Conversation generation using generative AI

[0176] Based on the collected personal data, the server uses generative AI to generate conversations that mimic the characteristics of deceased people and celebrities. Specifically, it uses OpenAI's API to generate conversations based on prompt sentences.

[0177] Speech recognition / synthesis means

[0178] The server uses a speech synthesis engine (e.g., pyttsx3) to convert the generated conversation into audio and synthesize a voice that is close to the voice of the deceased person or celebrity, allowing users to experience the conversation with a realistic voice.

[0179] Image recognition / synthesis methods

[0180] The server uses image processing libraries (e.g., OpenCV) to generate 3D models from photos and videos of deceased people and celebrities and synthesize realistic images, allowing users to experience realistic interactions.

[0181] Display means for realizing dialogue in virtual space

[0182] The device displays the generated conversations and images in a virtual space, allowing users to have the experience of interacting with deceased people or celebrities. Specifically, the device uses a smartphone and a head-mounted display to realize the interaction in the virtual space.

[0183] Examples

[0184] For example, consider the case where a user uploads a letter from their grandparents to an app and experiences the generated conversation. The user can talk to their grandparents in a virtual space using a smartphone or a head-mounted display. In this case, the following prompt sentence is used:

[0185] Examples of prompt statements

[0186] "Generate conversations by mimicking your grandparents' traits: Grandparents' letters, diaries, and family testimonials."

[0187] In this way, consumers can have a more realistic conversation experience with deceased people or celebrities.

[0188] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0189] Step 1:

[0190] Users use the device to upload personal data (letters, diaries, testimonials, etc.) of deceased people or celebrities.

[0191] Input: Personal data such as letters, diaries, testimonies, etc.

[0192] Output: Personal data stored on the server

[0193] Specific operation: A user uploads personal data to the application using a smartphone or PC. The uploaded data is stored on the server.

[0194] Step 2:

[0195] The server analyzes the personal data and sends a prompt to the generative AI.

[0196] Input: Personal data stored on the server

[0197] Output: The prompt sent to the generative AI

[0198] Specific operation: The server analyzes the uploaded personal data and generates a prompt to send to the generative AI. For example, it generates a prompt such as, "Please generate a conversation by imitating the characteristics of my grandparents: letters and diaries from my grandparents, and testimonials from family members."

[0199] Step 3:

[0200] A generative AI generates a conversation based on the prompt.

[0201] Input: Prompt sent to the generative AI

[0202] Output: Generated conversation text

[0203] Specific operation: The generative AI (e.g., OpenAI's API) generates a conversation text based on the prompt sentence. The generated conversation text is returned to the server.

[0204] Step 4:

[0205] The server transmits the generated conversation text to a speech recognition / synthesis means to generate speech.

[0206] Input: Generated conversation text

[0207] Output: Generated audio data

[0208] Specific operation: The server sends the generated conversation text to a speech synthesis engine (e.g., pyttsx3) to generate a voice similar to the voice of the deceased or famous person. The generated voice data is stored on the server.

[0209] Step 5:

[0210] The server sends photos of the deceased person or celebrity to an image recognition / synthesis tool, which generates a 3D model.

[0211] Input: A photo of a deceased or famous person

[0212] Output: The generated 3D model

[0213] How it works: The server sends photos of the deceased person or celebrity to an image processing library (e.g., OpenCV) to generate a 3D model, which is then stored on the server.

[0214] Step 6:

[0215] The device receives the generated voice data and 3D model and displays it in the virtual space.

[0216] Input: Generated voice data, generated 3D model

[0217] Output: Interactive experience in virtual space

[0218] Specific operation: The device (smartphone or head-mounted display) uses the voice data and 3D model received from the server to display a conversation with the deceased person or celebrity in a virtual space, allowing the user to have a realistic conversation experience.

[0219] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0220] "Example 1"

[0221] In one embodiment of the present invention, a system is provided for enabling general consumers to fulfill their desire to converse with celebrities or deceased people. The system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and an emotion engine. The emotion engine recognizes the user's emotions, and the generative AI generates a conversation in response to the emotions. Specifically, when the user is feeling joy, the generative AI generates a conversation in which the celebrity or the deceased person shares the joy. Also, when the user is feeling sad, the generative AI generates a conversation in which the celebrity or the deceased person comforts the user.

[0222] "Example 2"

[0223] In another embodiment of the present invention, a system is provided for enabling general consumers to fulfill their desire to converse with celebrities or deceased people. The system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and an emotion engine. The emotion engine recognizes the user's emotion, and the voice recognition / synthesis means and the image recognition / synthesis means generate voice and image in response to the emotion. Specifically, when the user feels surprised, the voice recognition / synthesis means generates a voice in which the celebrity or deceased makes a surprised voice, and the image recognition / synthesis means generates an image in which the celebrity or deceased looks surprised. Also, when the user feels relieved, the voice recognition / synthesis means generates a voice in which the celebrity or deceased makes a relieved voice, and the image recognition / synthesis means generates an image in which the celebrity or deceased looks relieved.

[0224] The process flow of each embodiment will be described below.

[0225] "Example 1"

[0226] Step 1: The user boots the system.

[0227] Step 2: The system collects the user's personal data.

[0228] Step 3: The emotion engine recognizes the user's emotion.

[0229] Step 4: The generative AI generates a conversation based on the user's emotions.

[0230] Step 5: The speech recognition / synthesis means outputs the generated conversation as audio.

[0231] Step 6: The image recognition / synthesis means outputs an image corresponding to the generated conversation.

[0232] "Example 2"

[0233] Step 1: The user boots the system.

[0234] Step 2: The system collects the user's personal data.

[0235] Step 3: The emotion engine recognizes the user's emotion.

[0236] Step 4: The voice recognition / synthesis means and the image recognition / synthesis means generate voice and images according to the user's emotions.

[0237] Step 5: The generated audio and images are presented to the user.

[0238] Example 1

[0239] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0240] In a system that allows ordinary consumers to fulfill their wish to talk to celebrities or deceased people, conventional technology has difficulty generating conversations according to the user's emotions, and has been unable to provide a more natural and emotionally sensitive dialogue experience. In addition, real-time processing is difficult when generating voice and images, limiting the user experience.

[0241] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0242] In this invention, the server includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and a conversation generation means according to emotion. This enables natural conversation generation according to the user's emotion, and provides a richer conversation experience through real-time voice and image generation.

[0243] "Personal data collection means" refers to means for collecting data related to specific individuals, such as publicly available information and content posted on social networking services.

[0244] "Generative AI" is an artificial intelligence technology for generating natural conversations based on collected data.

[0245] The "voice recognition / synthesis means" refers to a means including a voice recognition technology that converts voice data into text data and a voice synthesis technology that converts text data into voice data.

[0246] "Image recognition / synthesis means" refers to means including image recognition technology that analyzes image data and extracts specific information, and image synthesis technology that generates a new image.

[0247] The "emotion recognition means" is a means for analyzing and recognizing the user's emotions from the user's voice, facial expressions, etc.

[0248] The "means for generating conversation according to emotions" refers to a means for generating appropriate conversation content based on the recognized emotions of the user.

[0249] This invention is a system for fulfilling the desire of ordinary consumers to converse with celebrities or the deceased, and includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and an emotion-based conversation generation means.

[0250] Server Operation

[0251] The server first uses the personal data collection means to collect personal data such as public information of celebrities and deceased people and posts on social networking services. This collection is carried out using a web crawler or an API. For example, the server uses the API to collect past posts of celebrities.

[0252] The server then pre-processes the collected personal data, filtering out unnecessary information and cleaning up the text data, removing advertisements and spam from collected posts, and normalizing the text using natural language processing techniques.

[0253] The preprocessed data is then input into a generative AI system to generate conversations that mimic the characteristics of celebrities and deceased individuals. A common natural language generation technique is used as the generative AI model for this generation. For example, the server inputs a prompt sentence such as "How was your day today?" and generates a response from the celebrity that reads, "Today was a very good day. My new project was a success."

[0254] The generated speech is then converted to audio using speech recognition / synthesis means, such as using text-to-speech software. For example, the server may send the generated text to a text-to-speech API to generate an audio file.

[0255] Additionally, the server may use image recognition / synthesis means to generate an image of the celebrity or deceased person, using image generation technology, for example, the server may use image generation technology to generate a facial image of the celebrity and create an image file for display to the user.

[0256] Using the emotion recognition means, the server recognizes the user's emotions. It sends the user's voice and facial expression data to the emotion recognition API and judges whether the user is happy or sad.

[0257] Finally, the server uses generative artificial intelligence to generate appropriate conversations based on the user's recognized emotions. For example, if the user is sad, the server generates comforting conversations such as, "It's okay, you're not alone. I've been through similar experiences."

[0258] Terminal operation

[0259] The device receives voice and text input from the user and sends it to the server. For example, if a user says to a smartphone, "How was your day today?", the voice data is sent to the server.

[0260] When the device receives the generated conversation, image, and audio data from the server, it displays and plays it for the user. For example, the device displays the generated image of a celebrity on the screen and plays the audio from a speaker.

[0261] User Actions

[0262] Users can access the system through a terminal and enjoy the experience of conversing with celebrities or deceased people. For example, if a user starts the app on their smartphone and speaks to the terminal, asking "How was your day today?", the user can experience a dialogue with a celebrity or deceased person through the conversation, images, and audio returned from the terminal.

[0263] Examples

[0264] The user speaks to the device, saying, "How was your day today?" The device sends the speech to the server. The server uses generative artificial intelligence to generate a conversation in which the celebrity or deceased person says something like, "It was a great day today. My new project was a success." The generated conversation is converted to voice using speech recognition / synthesis means and sent to the device. The device plays the voice to the user.

[0265] Examples of prompt statements

[0266] "Generate how a celebrity would respond if a user asked, 'How was your day?'"

[0267] In this way, users can experience interactions with celebrities and deceased people.

[0268] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0269] Step 1:

[0270] The server uses the personal data collection means to collect personal data such as public information of celebrities and deceased people and content posted on social networking services. Specifically, the server acquires data using a web crawler or an API. For example, the server uses an API to collect past posts of celebrities. The input is the URL or API endpoint to be collected, and the output is the collected text data.

[0271] Step 2:

[0272] The server preprocesses the collected personal data. It filters unnecessary information and cleans the text data. Specifically, it removes advertisements and spam from the collected posts and normalizes the text using natural language processing techniques. The input is the collected text data, and the output is the preprocessed clean text data.

[0273] Step 3:

[0274] The server inputs the preprocessed data into a generative AI system to generate a conversation that imitates the characteristics of celebrities or deceased people. A common natural language generation technique is used as the generative AI model. For example, the server inputs a prompt sentence such as "How was your day today?" and generates a response from the celebrity such as "Today was a very good day. My new project was successful." The input is the prompt sentence and the preprocessed text data, and the output is the generated conversation text.

[0275] Step 4:

[0276] The server converts the generated dialogue into audio using speech recognition / synthesis means. For this conversion, the server uses voice synthesis software. For example, the server sends the generated text to a speech synthesis API to generate an audio file. The input is the generated dialogue text and the output is the audio file.

[0277] Step 5:

[0278] The server uses image recognition / synthesis means to generate images of celebrities and deceased individuals. The image generation uses image generation technology. For example, the server uses image generation technology to generate facial images of celebrities and create image files to display to users. The input is the generated conversation text and related image data, and the output is the generated image file.

[0279] Step 6:

[0280] The server recognizes the user's emotions using emotion recognition means. It sends the user's voice and facial expression data to the emotion recognition API and judges whether the user is happy or sad. The input is the user's voice data and facial expression images, and the output is the recognized emotion data.

[0281] Step 7:

[0282] The server uses generative artificial intelligence to generate appropriate conversations according to the user's recognized emotions. For example, if the user is sad, the server generates a comforting conversation such as, "Don't worry, you're not alone. I've been through similar experiences." The input is the recognized emotion data and a prompt sentence, and the output is a conversation text according to the emotion.

[0283] Step 8:

[0284] The terminal receives voice and text input from the user and sends it to the server. For example, when a user speaks to a smartphone, "How was your day today?", the voice data is sent to the server. The input is the user's voice and text input, and the output is the transmission of data to the server.

[0285] Step 9:

[0286] The terminal receives the generated conversation, image, and audio data from the server. For example, the terminal displays the generated image of a celebrity on the screen and plays the audio from a speaker. The input is the data from the server, and the output is the display and playback to the user.

[0287] Step 10:

[0288] Users access the system through a terminal and enjoy the experience of conversing with celebrities or deceased people. For example, if a user starts the app on their smartphone and speaks to the terminal, asking "How was your day today?", the user can experience a dialogue with a celebrity or deceased person through the conversation, images, and audio returned from the terminal. The input is the user's operation, and the output is the dialogue experience.

[0289] (Application example 1)

[0290] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0291] Although there are systems that allow ordinary consumers to fulfill their desire to converse with celebrities or deceased people, these systems have the problem that they cannot provide dialogue that responds to the user's emotions or an interactive experience in a virtual store. In addition, there is a lack of technology that can track the user's location and movement to enable dialogue with celebrities or deceased people in real time. This may limit the user experience and reduce satisfaction.

[0292] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0293] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion engine, an interactive display means for interacting with celebrities and deceased people in a virtual store, and a location recognition means for tracking the user's location and movement. This enables conversations according to the user's emotions and interactive experiences in the virtual store, improving the user experience and satisfaction.

[0294] "Means of collecting personal data" refers to means of collecting personal data such as publicly available information about celebrities or the deceased, or the contents of social media posts.

[0295] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of celebrities or deceased people based on collected personal data.

[0296] "Speech recognition / synthesis means" refers to a means for recognizing a user's utterances and converting the generated conversation into voice.

[0297] "Image recognition / synthesis means" means means for generating images of celebrities or deceased persons and tracking the user's location and movements.

[0298] An "emotion engine" is a means by which a generative AI recognizes the user's emotions and generates conversations based on those emotions.

[0299] An "interactive display means for interacting with celebrities or deceased people in a virtual store" is a means for displaying celebrities or deceased people in a virtual store and interactively interacting with the user.

[0300] "Location awareness means for tracking a user's location and movements" means means for recognizing a user's location and movements in real time and providing an interactive experience.

[0301] The system for implementing this invention is configured using the following hardware and software. A smartphone and a head-mounted display (HMD) are used as the hardware. Unity (game engine), TENSORFLOW (registered trademark) (generative AI model), Google Cloud Speech-to-Text (voice recognition), Google Cloud Text-to-Speech (voice synthesis), and OpenCV (image recognition) are used as the software.

[0302] The server includes personal data collection means, conversation generation means using generative AI, voice recognition / synthesis means, image recognition / synthesis means, an emotion engine, interactive display means for interacting with celebrities and deceased people in the virtual store, and location recognition means for tracking the user's location and movements.

[0303] First, the personal data collection means collects personal data such as public information of celebrities and deceased people and their social media posts. The collected data is analyzed using natural language processing (NLP) technology to extract important keywords and phrases.

[0304] Next, a generative AI-based conversation generator generates conversations that mimic the characteristics of celebrities or deceased people based on the collected personal data. The emotion engine adjusts the conversations to the user's emotions. For example, if the user is feeling happy, the generative AI generates conversations in which the celebrity or deceased person shares their joy.

[0305] The speech recognition / synthesis method uses Google Cloud Speech-to-Text to convert user speech into text in real time, and then converts the generated conversation into audio using Google Cloud Text-to-Speech.

[0306] The image recognition / synthesis method uses OpenCV to track the user's position and movements, generate 3D models of celebrities and deceased people within Unity, and display them interactively in response to the user's movements.

[0307] As a concrete example, when a user approaches the music section of a virtual store, a deceased famous musician appears and begins a conversation about the musician's life and music. When the user asks, "What is your favorite song?", the generative AI responds, "My favorite song is 'XX'," based on information obtained from the musician's interviews and social media posts.

[0308] An example of a prompt sentence is as follows:

[0309] User: What is your favorite song?

[0310] Generative AI model: My favorite song is "XX". This song has a special meaning in my life.

[0311] In this way, users can enjoy interacting with celebrities and deceased people within the virtual store.

[0312] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0313] Step 1:

[0314] The server uses the personal data collection means to collect personal data such as publicly available information and social media posts about celebrities and deceased people. As input, a web scraping tool (e.g., BeautifulSoup) is used to analyze the collected data using natural language processing (NLP) techniques to extract important keywords and phrases. As output, the analyzed personal data is obtained.

[0315] Step 2:

[0316] The server uses a generative AI-based conversation generation means to generate a conversation that mimics the characteristics of celebrities and deceased people based on the personal data collected in step 1. As input, the analyzed personal data is used to generate a conversation with a generative AI model (e.g., TensorFlow). As output, the generated conversation text is obtained.

[0317] Step 3:

[0318] The server uses an emotion engine to recognize the user's emotion and adjust the conversation generated in step 2 according to the user's emotion. As input, the server uses the user's emotion data and the generated conversation text to adjust the conversation in the emotion engine. As output, the adjusted conversation text is obtained.

[0319] Step 4:

[0320] The device uses speech recognition / synthesis to convert the user's speech into text in real time. As input, it uses the user's voice data and converts it into text using Google Cloud Speech-to-Text. As output, it obtains the text of the user's speech.

[0321] Step 5:

[0322] The server synthesizes the adjusted conversation text from step 3. As input, it uses the adjusted conversation text to generate speech data using Google Cloud Text-to-Speech. As output, it obtains the generated speech data.

[0323] Step 6:

[0324] The device tracks the user's location and movement using image recognition / synthesis. As input, it uses camera data and recognizes the user's location and movement using OpenCV. As output, it obtains the user's location data.

[0325] Step 7:

[0326] The server uses Unity to generate 3D models of celebrities and deceased people based on the user's location data obtained in step 6, and displays them interactively in the virtual store. The server uses the user's location data and the 3D model data of celebrities and deceased people as inputs, and generates interactive displays in Unity. The output is an interactive display in the virtual store.

[0327] Step 8:

[0328] Users enjoy interacting with celebrities and deceased people in a virtual store. As input, the system uses the user's speech and movements to generate dialogue and interact with the person through interactive displays. As output, the user's satisfaction is improved.

[0329] Example 2

[0330] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0331] In a system that allows ordinary consumers to fulfill their wish to talk to celebrities or deceased people, conventional technology had the problem that it was not possible to generate conversations or synthesize voice and images in real time, and it was not possible to respond according to the user's emotions. In addition, there was also the problem that the generated conversations, voices, and images were not natural, limiting the user experience.

[0332] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0333] In this invention, the server includes a personal data collection means, a conversation generation means using a generative AI model, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and an emotion-based voice and image generation means, which enables real-time conversation generation according to the user's emotions and the provision of natural voice and images.

[0334] "Personal data collection means" refers to a means for converting personal data provided by a user, such as letters, diaries, testimonies, etc., into digital format and storing it on a server.

[0335] A "generative AI model" is an artificial intelligence model that generates conversations that mimic the characteristics of a specific person based on collected personal data.

[0336] A "conversation generation means" is a means for generating a conversation based on collected personal data using a generative AI model.

[0337] The "voice recognition / synthesis means" refers to a means for converting the generated conversation data into voice and generating a voice that resembles the voice of a specific person.

[0338] An "image recognition / synthesis means" is a means for generating a 3D model from photographic data of a specific person and generating an image of that person.

[0339] The "emotion recognition means" is a means for analyzing the user's voice and facial expressions and recognizing the user's emotions.

[0340] The "means for generating voice and images according to emotions" refers to means for generating appropriate voice and images according to the user's emotions recognized by the emotion recognition means.

[0341] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0342] This invention is a system for enabling ordinary consumers to fulfill their desire to converse with celebrities or deceased people. This system includes a personal data collection means, a conversation generation means using a generative AI model, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and an emotion-based voice and image generation means.

[0343] Personal data collection methods

[0344] Users collect personal data such as past letters from grandparents or celebrities, diaries, and testimonials from family members. These data are converted into digital format using hardware such as scanners and cameras and stored on the terminal. The terminal then uploads the stored digital data to the server.

[0345] Conversation generation using generative AI models

[0346] The server receives the uploaded personal data and stores it in a database. It then uses a generative AI model (e.g., a general-purpose natural language processing model) to generate a conversation based on the personal data. This generative AI model analyzes the collected data and generates a natural conversation that mimics the characteristics of a specific person.

[0347] Speech recognition / synthesis means

[0348] The server sends the generated conversation data to a voice recognition / synthesis means (e.g., general-purpose voice synthesis software). The voice recognition / synthesis means converts the conversation data into a voice that resembles the voice of a specific person. The generated voice data is temporarily stored on the server.

[0349] Image recognition / synthesis methods

[0350] The server sends photo data of a specific person to an image recognition / synthesis means (e.g., general-purpose image generation technology). The image recognition / synthesis means generates a 3D model from the photo data and creates an image of the person. The generated image data is temporarily stored on the server.

[0351] emotion recognition means

[0352] While the user is using the system, the server transmits the user's voice and facial expression in real time to an emotion recognition means (e.g., general-purpose emotion analysis software). The emotion recognition means analyzes the user's emotions and sends feedback to the server.

[0353] Means for generating sounds and images according to emotions

[0354] The server issues instructions to the voice recognition / synthesis means and the image recognition / synthesis means based on feedback from the emotion recognition means to generate appropriate voice and image. For example, if the user feels surprised, the voice recognition / synthesis means generates a surprised voice, and the image recognition / synthesis means generates a surprised image.

[0355] Examples

[0356] Example 1: Conversation with grandparents

[0357] If a user wants to experience a conversation with their grandparents, they can input the following prompt into the generative AI model:

[0358] "Grandpa, please tell me some of your memories from the past."

[0359] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the grandparents' characteristics. It then uses a speech recognition / synthesis means to generate a voice that resembles the grandparents' voice, and an image recognition / synthesis means to generate an image of the grandparents.

[0360] Example 2: Conversation with a celebrity

[0361] If a user wants to experience a conversation with a celebrity, they can input the following prompt into the generative AI model:

[0362] "Celebrities, tell us about your accomplishments."

[0363] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the celebrity's characteristics, then uses a speech recognition / synthesis method to generate a voice similar to the celebrity's voice, and an image of the celebrity is generated using an image recognition / synthesis method.

[0364] The system allows users to experience realistic conversations with deceased people or celebrities.

[0365] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0366] System program processing flow

[0367] Step 1: Collecting personal data

[0368] Users collect personal data such as past letters from grandparents or celebrities, diaries, testimonials from family members, etc. This data is converted into digital form using hardware such as scanners and cameras and stored on the device.

[0369] Input: Physical data such as letters, diaries, testimonies, etc.

[0370] Output: Personal data in digital format

[0371] Step 2: Upload your data

[0372] The terminal uploads the stored digital data to a server.

[0373] Input: Personal data in digital form

[0374] Output: Digital data stored on a server

[0375] Step 3: Conversation generation

[0376] The server receives the uploaded personal data, stores it in a database, and then uses a generative AI model to generate a conversation based on the personal data.

[0377] Input: Personal data in digital form

[0378] Output: Generated conversation data

[0379] Step 4: Text-to-speech

[0380] The server transmits the generated conversation data to a voice recognition / synthesis means, which converts the conversation data into a voice that resembles the voice of a specific person.

[0381] Input: Generated conversation data

[0382] Output: Generated audio data

[0383] Step 5: Image Generation

[0384] The server transmits photographic data of a particular person to the image recognition / synthesis means, which generates a 3D model from the photographic data and creates an image of the person.

[0385] Input: Photo data of a specific person

[0386] Output: Generated image data

[0387] Step 6: Emotion Recognition

[0388] While the user is using the system, the server transmits the user's voice and facial expression in real time to the emotion recognition means, which analyzes the user's emotions and sends feedback to the server.

[0389] Input: User's voice and facial expression data

[0390] Output: Parsed emotion data

[0391] Step 7: Emotion-based voice and image generation

[0392] The server issues instructions to the voice recognition / synthesis means and the image recognition / synthesis means based on feedback from the emotion recognition means to generate appropriate voice and image. For example, if the user feels surprised, the voice recognition / synthesis means generates a surprised voice, and the image recognition / synthesis means generates a surprised image.

[0393] Input: Parsed emotion data

[0394] Output: Voice data and image data according to emotions

[0395] Examples

[0396] Example 1: Conversation with grandparents

[0397] If a user wants to experience a conversation with their grandparents, they can input the following prompt into the generative AI model:

[0398] "Grandpa, please tell me some of your memories from the past."

[0399] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the grandparents' characteristics. It then uses a speech recognition / synthesis means to generate a voice that resembles the grandparents' voice, and an image recognition / synthesis means to generate an image of the grandparents.

[0400] Example 2: Conversation with a celebrity

[0401] If a user wants to experience a conversation with a celebrity, they can input the following prompt into the generative AI model:

[0402] "Celebrities, tell us about your accomplishments."

[0403] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the celebrity's characteristics, then uses a speech recognition / synthesis method to generate a voice similar to the celebrity's voice, and an image of the celebrity is generated using an image recognition / synthesis method.

[0404] The system allows users to experience realistic conversations with deceased people or celebrities.

[0405] (Application example 2)

[0406] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0407] Conventional technology has made it difficult for ordinary consumers to experience conversations with deceased people or celebrities. In particular, providing a realistic conversation experience that corresponds to the user's emotions is technically complex and difficult to achieve. Furthermore, there has been no system that can easily provide such an experience using a mobile terminal such as a smartphone. As a result, users have been unable to enjoy a realistic conversation experience that corresponds to their emotions.

[0408] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0409] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion engine, a means for generating voice and images according to the user's emotion, and an application installed on a smartphone, which allows the user to experience a real conversation with a deceased person or a celebrity.

[0410] "Personal data collection means" are means for collecting personal information of deceased persons or celebrities, such as past letters or diaries, or testimonies from family members.

[0411] A "conversation generation method using generative AI" is a method that uses artificial intelligence technology to generate conversations that imitate the characteristics of a deceased person or celebrity based on collected personal data.

[0412] The "voice recognition / synthesis means" refers to a means for recognizing the user's voice and converting the generated conversation into voice.

[0413] "Image recognition / synthesis means" means means for generating images using 3D models generated from photographs of deceased persons or celebrities.

[0414] An "emotion engine" is a means for recognizing a user's emotions and generating sounds and images according to those emotions.

[0415] "Means for generating voice and images according to the user's emotions" refers to means for controlling voice recognition / synthesis means and image recognition / synthesis means based on the emotions felt by the user, and generating appropriate voice and images.

[0416] The "application installed on a smartphone" is an application that runs on a smartphone and allows the user to enjoy the experience of having a conversation with a deceased person or a celebrity.

[0417] The system for implementing this invention uses the following hardware and software. The hardware requires a smartphone, a microphone, and a speaker. The software uses Python, OpenAI API, Google Text-to-Speech (gTTS), OpenCV, and SpeechRecognition.

[0418] The server uses a personal data collection means to collect personal information of the deceased or famous person, such as past letters and diaries, testimonials from family members, etc. This collected data is analyzed by a conversation generation means using generative AI, and conversations that imitate the characteristics of the deceased or famous person are generated.

[0419] The terminal (smartphone) uses a voice recognition / synthesis means to recognize the user's voice and convert the generated conversation into audio.Furthermore, it uses an image recognition / synthesis means to generate an image using a 3D model generated from a photograph of a deceased person or a celebrity.

[0420] The emotion engine recognizes the user's emotion and generates voice and image according to the emotion. For example, if the user feels surprised, the voice recognition / synthesis means generates a voice that sounds surprised, and the image recognition / synthesis means generates an image that looks surprised. On the other hand, if the user feels relieved, the voice recognition / synthesis means generates a voice that sounds relieved, and the image recognition / synthesis means generates an image that looks relieved.

[0421] As a concrete example, if a user says, "What happened today?", the emotion engine recognizes the user's emotion as "surprise." In this case, the generative AI generates a conversation using the following prompt sentence:

[0422] Example prompt:

[0423] “Generate conversations when users have an emotion of surprise: What happened today?”

[0424] By inputting this prompt sentence into the generative AI model, a conversation corresponding to the emotion of surprise is generated. The generated conversation is converted into voice by the speech recognition / synthesis means, and an image with a surprised facial expression is generated by the image recognition / synthesis means.

[0425] As described above, the present invention is a system that enables a user to experience a realistic conversation with a deceased person or a celebrity.

[0426] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0427] Step 1:

[0428] The server uses the personal data collection means to collect personal information such as past letters and diaries of deceased people and celebrities, and testimonials from family members. The input is personal data such as letters, diaries, and testimonials, and the output is a database of the collected personal data. Specifically, the server saves the data in the database and organizes and classifies the data as necessary.

[0429] Step 2:

[0430] The server analyzes the collected personal data using a generative AI-based conversation generation means to generate conversations that mimic the characteristics of deceased people or celebrities. The input is the collected personal data, and the output is the generated conversation text. Specifically, the server inputs a prompt sentence into the generative AI model to generate the conversation text.

[0431] Step 3:

[0432] The terminal (smartphone) recognizes the user's voice using a speech recognition / synthesis means. The input is the user's voice, and the output is the recognized text. Specifically, the voice is captured using a microphone and converted to text using the SpeechRecognition library.

[0433] Step 4:

[0434] The terminal uses an emotion engine to recognize the user's emotion. The input is the recognized text, and the output is the recognized emotion. Specifically, the terminal performs text analysis and estimates the emotion.

[0435] Step 5:

[0436] The server uses generative AI to generate a conversation according to the user's emotions. The input is the recognized emotion and the user's text, and the output is a conversation text according to the emotion. Specifically, it generates a prompt sentence and inputs it into the generative AI model to generate the conversation text. An example of a prompt sentence is "Generate a conversation when the user has an emotion of surprise: What happened today?"

[0437] Step 6:

[0438] The terminal uses a speech recognition / synthesis means to convert the generated conversation text into speech. The input is the generated conversation text, and the output is an audio file. Specifically, the terminal uses Google Text-to-Speech (gTTS) to convert the text into speech and generate an audio file.

[0439] Step 7:

[0440] The device uses image recognition / synthesis to generate images using 3D models generated from photos of deceased people or celebrities. The input is the recognized emotion and the 3D model, and the output is an image corresponding to the emotion. Specifically, the device uses OpenCV to generate an image with a facial expression corresponding to the emotion.

[0441] Step 8:

[0442] The terminal provides the generated audio and images to the user. The input is an audio file and an image, and the output is the audio and images provided to the user. In concrete terms, the terminal operates by playing the audio using a speaker and displaying the images using a display.

[0443] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0444] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0445] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0446] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0447] [Second embodiment]

[0448] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0449] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0450] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0451] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0452] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0453] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0454] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0455] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0456] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0457] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0458] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0459] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0460] "Example 1"

[0461] As one embodiment of the present invention, a system is provided that allows general consumers to experience conversations with celebrities and deceased persons. This system is configured by combining a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, and an image recognition / synthesis means. The personal data collection means collects personal data such as public information of celebrities and deceased persons and SNS posts. The generative AI generates a conversation that imitates the characteristics of the celebrity or deceased person based on the collected personal data. The voice recognition / synthesis means converts the generated conversation into voice. The image recognition / synthesis means generates an image of the celebrity or deceased person. By combining these means, a system is realized that allows general consumers to experience conversations with celebrities and deceased persons.

[0462] "Example 2"

[0463] As a specific example, consider a system that allows general consumers to experience a conversation with their deceased grandparents. First, a personal data collection means collects personal data such as the grandparents' past letters and diaries, and testimonies from family members. A generative AI generates conversation that mimics the characteristics of the grandparents based on the collected personal data. A voice recognition / synthesis means synthesizes the generated conversation into a voice that is close to the grandparents' voice. An image recognition / synthesis means generates images of the grandparents using 3D models generated from photos of the grandparents. By combining these means, a system is realized that allows general consumers to experience a conversation with their deceased grandparents.

[0464] The process flow of each embodiment will be described below.

[0465] "Example 1"

[0466] Step 1: Personal data such as publicly available information and social media posts about celebrities and deceased people is collected using personal data collection methods.

[0467] Step 2: Generative AI generates conversations that mimic the characteristics of celebrities or deceased people based on the collected personal data.

[0468] Step 3: The speech recognition / synthesis means converts the generated speech into audio.

[0469] Step 4: Image recognition / synthesis procedures generate an image of the celebrity or deceased person.

[0470] "Example 2"

[0471] Step 1: Personal data collection instruments are used to collect personal data such as grandparents' past letters and diaries, and testimonies from family members.

[0472] Step 2: The generative AI generates conversations that mimic the characteristics of the grandparents based on the collected personal data.

[0473] Step 3: The speech recognition / synthesis means synthesizes the generated conversation into a voice that resembles the grandparents' voice.

[0474] Step 4: The image recognition / synthesis means generates images of the grandparents using the 3D models generated from the photos of the grandparents.

[0475] Example 1

[0476] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0477] There is a demand for systems that allow ordinary consumers to fulfill their desire to talk to celebrities or deceased people to provide a more realistic and natural conversation experience. However, conventional systems have limitations in the quality of collected data, the naturalness of generated conversations, and the realism of voice and images, resulting in an insufficient user experience.

[0478] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0479] In this invention, the server includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition means, a voice synthesis means, an image recognition means, and an image generation means, which enable the server to generate a conversation that imitates the characteristics of a specific person based on the collected personal data, convert a user's voice input into text, vocalize the generated conversation, and recognize and generate an image of the specific person.

[0480] "Personal data collection means" refers to means for collecting data such as publicly available information about a specific person and the contents of social media posts.

[0481] "Generative AI" is AI that generates conversations that mimic the characteristics of a specific person based on collected data.

[0482] A "speech recognition means" is a means for converting a user's voice input into text.

[0483] The "voice synthesis means" is a means for converting the generated conversation into voice.

[0484] The "image recognition means" is a means for recognizing an image of a specific person.

[0485] An "image generating means" is a means for generating an image of a particular person.

[0486] This invention is a system for fulfilling the desire of ordinary consumers to converse with celebrities or deceased people, and includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition means, a voice synthesis means, an image recognition means, and an image generation means.

[0487] Personal data collection methods

[0488] The server collects personal data such as public information and social media posts of celebrities and deceased people. This data is collected using web scraping tools (e.g., Beautiful Soup and Scrapy) and social media APIs (e.g., Twitter API and Facebook Graph API). The collected data is stored in a database.

[0489] Conversation Generation Method

[0490] The server generates a conversation using generative artificial intelligence (e.g., OpenAI's GPT-4) based on the collected personal data. The generative artificial intelligence analyzes the input personal data and generates natural conversational sentences. An example of a prompt sentence is, "Based on the public information and SNS posts of famous singer A, please generate a conversation that imitates his characteristics."

[0491] Speech recognition and synthesis means

[0492] The server converts the generated conversational text into voice using speech synthesis software (e.g., Google Text-to-Speech or Amazon Polly). It also converts the user's voice input into text using speech recognition software (e.g., Google Speech-to-Text or IBM Watson Speech to Text). This allows the user's question to be obtained as text and input into the generative artificial intelligence.

[0493] Image recognition and image generation

[0494] The server uses image generation software (e.g. DALL-E or DeepArt) to generate images of celebrities or deceased people, allowing the user to visually recognize the person they are speaking with. It inputs prompts to the image generation software based on the collected personal data, and sends the generated image to the user's device.

[0495] User interaction

[0496] Users access the system through a terminal (e.g., a smartphone or PC) and experience conversations with celebrities or deceased people. When the user asks a question by voice, the voice recognition means converts the question into text, and the generative artificial intelligence generates an answer. The generated answer is converted into voice by the voice synthesis means and returned to the user. At the same time, an image of the celebrity or deceased person, generated by the image recognition means and the image generation means, is displayed.

[0497] Examples

[0498] For example, if a user wishes to "talk to famous singer A," the system operates as follows.

[0499] 1. The server collects public information and social media posts about famous singer A.

[0500] 2. The server uses generative artificial intelligence based on the collected data to generate conversations that mimic the characteristics of famous singer A.

[0501] 3. The server converts the generated conversational text into voice using speech synthesis software.

[0502] 4. The server generates an image of famous singer A.

[0503] 5. The user experiences a conversation with famous singer A through the terminal using audio and images.

[0504] In this way, a system is realized that allows ordinary consumers to experience conversations with celebrities and deceased people.

[0505] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0506] Step 1:

[0507] The server uses personal data collection means to collect public information and social media posts about celebrities and deceased people. Specifically, it uses web scraping tools (e.g. Beautiful Soup and Scrapy) and social media APIs (e.g. Twitter API and Facebook Graph API). The input is the name and related information of a specific person, and the output is the collected personal data. The collected data is stored in a database.

[0508] Step 2:

[0509] The server retrieves the collected personal data from the database and generates a conversation using generative artificial intelligence (e.g., OpenAI's GPT-4). The input is the collected personal data, and the output is the generated conversation. Specifically, a prompt sentence is input into the generative artificial intelligence, and a conversation sentence that imitates the characteristics of a specific person is generated. An example of a prompt sentence is, "Based on the public information of famous singer A and his social media posts, please generate a conversation that imitates his characteristics."

[0510] Step 3:

[0511] The server converts the generated conversation text into voice using speech synthesis software (e.g., Google Text-to-Speech or Amazon Polly). The input is the generated conversation text, and the output is an audio file. Specifically, the generated conversation text is input into the speech synthesis software to generate an audio file.

[0512] Step 4:

[0513] The server converts the user's voice input into text using voice recognition software (e.g., Google Speech-to-Text or IBM Watson Speech to Text). The input is the user's voice, and the output is text. Specifically, the user's voice input is input into the voice recognition software and converted into text.

[0514] Step 5:

[0515] The server generates an image of a specific person using image generation software (e.g., DALL-E or DeepArt) based on the collected personal data. The input is the collected personal data, and the output is the generated image. Specifically, a prompt is input to the image generation software, and an image of a specific person is generated.

[0516] Step 6:

[0517] Users access the system through a terminal (e.g., a smartphone or PC) and experience a conversation with a celebrity or a deceased person. The input is the user's voice question, and the output is the generated audio and image of the conversation. Specifically, when the user asks a question by voice, the voice recognition means converts the question into text, and the generative artificial intelligence generates an answer. The generated answer is converted into voice by the voice synthesis means and returned to the user. At the same time, an image of the celebrity or deceased person, generated by the image recognition means and the image generation means, is displayed.

[0518] (Application example 1)

[0519] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0520] When ordinary consumers wish to converse with celebrities or deceased people, it is difficult to provide a realistic experience that would allow them to realize their wish. In particular, advanced technology is required to provide a real-time conversation experience that combines voice and images. There is also a demand to provide this experience in a form that is easy for users to use.

[0521] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0522] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, an application means installed on a smartphone, a means for recognizing a user's voice input and generating a voice response in real time, and a means for generating an image of a celebrity or a deceased person and displaying it during a conversation. This allows general consumers to easily enjoy a real conversation experience with a celebrity or a deceased person.

[0523] "Personal data collection means" refers to means for collecting personal data, such as publicly available information about celebrities or the deceased, or the content of social media posts.

[0524] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of celebrities or deceased people based on collected personal data.

[0525] "Speech recognition / synthesis means" means for recognizing a user's voice input and converting the generated speech into voice.

[0526] "Image recognition / synthesis means" refers to means for generating images of celebrities or deceased people and displaying them during a conversation.

[0527] The "application means installed on a smartphone" refers to a means that is installed on a smartphone and provides an application that enables a user to have a conversation experience with a celebrity or a deceased person.

[0528] The "means for recognizing a user's voice input and generating a voice response in real time" refers to a means for recognizing a user's voice input in real time and generating a voice response in response thereto.

[0529] "Means for generating an image of a celebrity or a deceased person and displaying it during a conversation" refers to means for generating an image of a celebrity or a deceased person and displaying it to a user during a conversation.

[0530] The system for carrying out the present invention is configured as follows. First, the server uses a personal data collection means to collect personal data such as public information and social media posts of celebrities and deceased people. This collected data is input to a conversation generation means using generative AI, and conversations that imitate the characteristics of celebrities and deceased people are generated.

[0531] Next, an application installed on the user's device (smartphone) uses speech recognition / synthesis means to recognize the user's voice input and generates a real-time voice response to convert the generated conversation into voice. It also uses image recognition / synthesis means to generate images of celebrities and deceased people and display them to the user during the conversation.

[0532] Specifically, the server uses the following software and hardware: It uses the OpenAI API for generative AI, the SpeechRecognition library for voice recognition, and the gTTS library for voice synthesis. It uses the OpenCV library for image processing. These software run on the server and work in conjunction with applications installed on the user's smartphone.

[0533] For example, if a user wants to talk to a "historical figure," the server collects personal data about that person and uses generative AI to generate a conversation that mimics the characteristics of that person. When the user asks a question to the smartphone, the voice recognition / synthesis means recognizes the question and generates a voice response in real time. At the same time, the image recognition / synthesis means generates an image of the person and displays it on the smartphone screen.

[0534] An example of a prompt is as follows:

[0535] "Collect personal data about a historical figure and generate answers to the following user questions that mimic the characteristics of that person.

[0536] User: What is your dream?

[0537] AI:

[0538] In this way, users can easily enjoy the experience of having a real conversation with a celebrity or a deceased person.

[0539] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0540] Step 1:

[0541] The server uses the personal data collection means to collect personal data such as public information and social media posts of celebrities and deceased individuals. The input is the name of the celebrity or deceased individual, and the output is the collected personal data. This data collection is done through public information on the Internet and social media APIs.

[0542] Step 2:

[0543] The server inputs the collected personal data into a generative AI to generate a conversation that imitates the characteristics of celebrities and deceased people. The input is the collected personal data, and the output is the generated conversation text. The generative AI uses OpenAI's API to generate a conversation based on the prompt sentence.

[0544] Step 3:

[0545] The user's device (smartphone) recognizes the user's voice input using a speech recognition / synthesis means. The input is the user's voice and the output is the user's question in text form. The SpeechRecognition library is used for speech recognition.

[0546] Step 4:

[0547] The server inputs the user's question text into the generative AI, which generates a voice response in real time. The input is the user's question text, and the output is the generated response text. The generative AI again uses OpenAI's API to generate a response based on the prompt text.

[0548] Step 5:

[0549] The user's terminal inputs the generated response text to a speech synthesis means and converts it into speech. The input is the generated response text, and the output is an audio file. The gTTS library is used for speech synthesis.

[0550] Step 6:

[0551] The server generates images of celebrities and deceased people using image recognition / synthesis methods. The input is the name of the celebrity or deceased person, and the output is the generated image file. The OpenCV library is used for image generation.

[0552] Step 7:

[0553] The user's terminal displays the generated audio and image files. The input is the audio and image files, and the output is visual and auditory feedback to the user. The audio files are played through the speaker, and the image files are shown on the display.

[0554] Example 2

[0555] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0556] Conventional technology has made it difficult for ordinary consumers to experience conversations with deceased people or specific individuals. In particular, generating natural conversations that reflect the characteristics of the deceased, and realistic voices and images, is technically complex and difficult to achieve. For this reason, there is a demand for a system that can reproduce conversations with the deceased.

[0557] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0558] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a means for providing a conversation experience by combining the generated voice and image, thereby enabling general consumers to experience a natural conversation with a deceased person or a specific person.

[0559] "Personal data collection means" refers to devices or methods for collecting and digitizing personal information of a specific individual, such as past letters, diaries, and testimonies.

[0560] A "conversation generation method using generative AI" is an artificial intelligence technology for generating conversations that mimic the characteristics of a specific individual based on collected personal data.

[0561] "Speech recognition / synthesis means" refers to technology for converting the generated conversation into voice data and generating a voice that resembles the voice of a specific person.

[0562] "Image recognition / synthesis means" is a technology for generating 3D models and realistic images based on photographs or videos of specific people.

[0563] "Means for combining generated audio and images to provide a conversational experience" refers to an apparatus or method for synchronizing generated audio data and image data to provide a user with an interactive conversational experience.

[0564] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0565] This invention is a system for enabling general consumers to experience conversations with deceased people or specific individuals. The system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a means for providing a conversation experience by combining the generated voice and image.

[0566] Collection of Personal Data

[0567] Users collect personal data such as past letters from grandparents or specific individuals, diaries, and testimonials from family members. These data are digitized using a scanner, voice recording device, and text input software. For example, a user can use a scanner to scan letters from grandparents and save them as text data.

[0568] Conversation generation using generative AI

[0569] The server receives the collected personal data and uses a generative AI model (e.g., a general natural language processing model) to generate a conversation that mimics the characteristics of grandparents or a specific person. Specifically, the server analyzes the data and inputs a prompt sentence into the generative AI model. For example, the server inputs a prompt sentence such as, "Based on the contents of the letter from my grandparents, please generate a conversation that reflects the characteristics of my grandparents."

[0570] Speech Recognition / Synthesis

[0571] The server uses a voice recognition / synthesis means (for example, a general voice synthesis API) to synthesize the generated conversation into a voice that resembles the voice of a grandparent or a specific person. Specifically, the server sends the generated text data to a voice synthesis API and obtains a voice file. For example, the generated conversation, "It's a very nice day today," is synthesized into the voice of a grandparent.

[0572] Image Recognition / Synthesis

[0573] The server uses image recognition / synthesis means (e.g., a general image generation algorithm) to generate a 3D model using photos of the grandparents or a specific person. Specifically, the server analyzes the photo data and inputs it into a 3D model generation algorithm. For example, a 3D model is generated based on a photo of the grandparents' faces, and an avatar that moves in real time is created.

[0574] Providing a conversational experience

[0575] The device combines the generated voice and image to provide the user with a conversation experience with a grandparent or a specific person. Specifically, the device plays the audio file and the 3D model in sync, allowing the user to have an interactive conversation. For example, the user can talk to an avatar of a grandparent while looking at the screen of the device.

[0576] Examples and prompts

[0577] Examples

[0578] A user scans a letter from their grandparents and uploads it to the system. The server analyzes the contents of the letter and uses a generative AI model to generate a conversation that mimics the characteristics of the grandparents. The generated conversation is then converted into voice using a common voice synthesis API, and a 3D model of the grandparent is generated using a common image generation algorithm. Finally, the user can talk to the grandparents through the device.

[0579] Examples of prompt statements

[0580] "Based on the content of the letter below, please generate a conversation that reflects the characteristics of the grandparents. The content of the letter is as follows: 'Dear Grandson, It's a very nice day today and the flowers in the garden are in full bloom. If you were here, we could go for a walk together.'"

[0581] By inputting this prompt into the generative AI model, natural conversation that reflects the characteristics of the grandparents is generated.

[0582] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0583] Step 1:

[0584] The user collects personal data such as past letters, diaries, and testimonials from family members of grandparents or other specific individuals. These data are digitized using a scanner, a voice recording device, and text input software. Specifically, the user uses a scanner to scan letters from grandparents and saves them as text data. The input is physical personal data such as letters and diaries, and the output is digitized text data.

[0585] Step 2:

[0586] The server receives digitized personal data uploaded by the user. It then uses a generative AI model (e.g., a general natural language processing model) to generate a conversation that mimics the characteristics of a specific person. Specifically, the server analyzes the data and inputs a prompt sentence into the generative AI model. For example, the server inputs the prompt sentence, "Based on the contents of my grandparents' letters, please generate a conversation that reflects my grandparents' characteristics." The input is the digitized personal data, and the output is the generated conversation text.

[0587] Step 3:

[0588] The server synthesizes the generated conversation text into a voice that resembles the voice of a specific person using a voice recognition / synthesis means (for example, a general voice synthesis API). Specifically, the server sends the generated text data to a voice synthesis API and obtains an audio file. For example, the generated conversation "It's a very nice day today" is synthesized in the voice of a grandparent. The input is the generated conversation text, and the output is an audio file.

[0589] Step 4:

[0590] The server uses image recognition / synthesis means (e.g., a general image generation algorithm) to generate a 3D model using a photo of a specific person. Specifically, the server analyzes the photo data and inputs it into a 3D model generation algorithm. For example, a 3D model is generated based on a photo of a grandparent's face, and an avatar that moves in real time is created. The input is photo data of a specific person, and the output is a 3D model.

[0591] Step 5:

[0592] The terminal combines the generated audio file with the 3D model to provide the user with a conversation experience with a specific person. Specifically, the terminal plays the audio file and the 3D model in sync, allowing the user to have an interactive conversation. For example, a user can interact with an avatar of their grandparents while looking at the screen of the terminal. The input is the audio file and the 3D model, and the output is an interactive conversation experience.

[0593] (Application example 2)

[0594] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0595] There are existing systems that allow ordinary consumers to fulfill their desire to converse with deceased relatives or celebrities, but these systems have the problem of lacking realism and limiting the user experience. In addition, there is a lack of means to realize dialogue in a virtual space, making it difficult to provide a more realistic experience. This has led to a demand for a system that allows users to experience a more realistic conversation with deceased relatives or celebrities.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0597] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a display means for realizing a conversation in a virtual space, which allows general consumers to experience a more realistic conversation with a deceased person or a celebrity.

[0598] "Personal data collection methods" are methods for collecting data about individuals, such as past letters, diaries, and testimonies from family members of deceased or famous people.

[0599] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of deceased people or celebrities based on collected personal data.

[0600] The "voice recognition / synthesis means" refers to a means for converting the generated conversation into voice and synthesizing a voice that is close to the voice of the deceased or famous person.

[0601] "Image recognition / synthesis means" refers to a means for generating 3D models from photographs or videos of deceased persons or celebrities and synthesizing realistic images.

[0602] "Display means for realizing dialogue in a virtual space" refers to a means for displaying generated conversations and images in a virtual space, providing a user with the experience of interacting with a deceased person or a celebrity.

[0603] The following system configuration will be described as an embodiment of the present invention.

[0604] System Configuration

[0605] This system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a display means for realizing dialogue in a virtual space.

[0606] Personal data collection methods

[0607] The server collects data about deceased or famous people, such as past letters, diaries, and testimonials from family members. This data may be provided by users or collected from public sources on the Internet.

[0608] Conversation generation using generative AI

[0609] Based on the collected personal data, the server uses generative AI to generate conversations that mimic the characteristics of deceased people and celebrities. Specifically, it uses OpenAI's API to generate conversations based on prompt sentences.

[0610] Speech recognition / synthesis means

[0611] The server uses a speech synthesis engine (e.g., pyttsx3) to convert the generated conversation into audio and synthesize a voice that is close to the voice of the deceased person or celebrity, allowing users to experience the conversation with a realistic voice.

[0612] Image recognition / synthesis methods

[0613] The server uses image processing libraries (e.g., OpenCV) to generate 3D models from photos and videos of deceased people and celebrities and synthesize realistic images, allowing users to experience realistic interactions.

[0614] Display means for realizing dialogue in virtual space

[0615] The device displays the generated conversations and images in a virtual space, allowing users to have the experience of interacting with deceased people or celebrities. Specifically, the device uses a smartphone and a head-mounted display to realize the interaction in the virtual space.

[0616] Examples

[0617] For example, consider the case where a user uploads a letter from their grandparents to an app and experiences the generated conversation. The user can talk to their grandparents in a virtual space using a smartphone or a head-mounted display. In this case, the following prompt sentence is used:

[0618] Examples of prompt statements

[0619] "Generate conversations by mimicking your grandparents' traits: Grandparents' letters, diaries, and family testimonials."

[0620] In this way, consumers can have a more realistic conversation experience with deceased people or celebrities.

[0621] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0622] Step 1:

[0623] Users use the device to upload personal data (letters, diaries, testimonials, etc.) of deceased people or celebrities.

[0624] Input: Personal data such as letters, diaries, testimonies, etc.

[0625] Output: Personal data stored on the server

[0626] Specific operation: A user uploads personal data to the application using a smartphone or PC. The uploaded data is stored on the server.

[0627] Step 2:

[0628] The server analyzes the personal data and sends a prompt to the generative AI.

[0629] Input: Personal data stored on the server

[0630] Output: The prompt sent to the generative AI

[0631] Specific operation: The server analyzes the uploaded personal data and generates a prompt to send to the generative AI. For example, it generates a prompt such as, "Please generate a conversation by imitating the characteristics of my grandparents: letters and diaries from my grandparents, and testimonials from family members."

[0632] Step 3:

[0633] A generative AI generates a conversation based on the prompt.

[0634] Input: Prompt sent to the generative AI

[0635] Output: Generated conversation text

[0636] Specific operation: The generative AI (e.g., OpenAI's API) generates a conversation text based on the prompt sentence. The generated conversation text is returned to the server.

[0637] Step 4:

[0638] The server transmits the generated conversation text to a speech recognition / synthesis means to generate speech.

[0639] Input: Generated conversation text

[0640] Output: Generated audio data

[0641] Specific operation: The server sends the generated conversation text to a speech synthesis engine (e.g., pyttsx3) to generate a voice similar to the voice of the deceased or famous person. The generated voice data is stored on the server.

[0642] Step 5:

[0643] The server sends photos of the deceased person or celebrity to an image recognition / synthesis tool, which generates a 3D model.

[0644] Input: A photo of a deceased or famous person

[0645] Output: The generated 3D model

[0646] How it works: The server sends photos of the deceased person or celebrity to an image processing library (e.g., OpenCV) to generate a 3D model, which is then stored on the server.

[0647] Step 6:

[0648] The device receives the generated voice data and 3D model and displays it in the virtual space.

[0649] Input: Generated voice data, generated 3D model

[0650] Output: Interactive experience in virtual space

[0651] Specific operation: The device (smartphone or head-mounted display) uses the voice data and 3D model received from the server to display a conversation with the deceased person or celebrity in a virtual space, allowing the user to have a realistic conversation experience.

[0652] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0653] "Example 1"

[0654] In one embodiment of the present invention, a system is provided for enabling general consumers to fulfill their desire to converse with celebrities or deceased people. The system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and an emotion engine. The emotion engine recognizes the user's emotions, and the generative AI generates a conversation in response to the emotions.

[0655] Specifically, if the user is feeling happy, the generative AI will generate a conversation in which a celebrity or deceased person shares the joy, and if the user is feeling sad, the generative AI will generate a conversation in which a celebrity or deceased person comforts the user.

[0656] "Example 2"

[0657] In another embodiment of the present invention, a system is provided for enabling general consumers to fulfill their desire to converse with celebrities or deceased people. The system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and an emotion engine. The emotion engine recognizes the user's emotion, and the voice recognition / synthesis means and the image recognition / synthesis means generate voice and image in response to the emotion. Specifically, when the user feels surprised, the voice recognition / synthesis means generates a voice in which the celebrity or deceased makes a surprised voice, and the image recognition / synthesis means generates an image in which the celebrity or deceased looks surprised. Also, when the user feels relieved, the voice recognition / synthesis means generates a voice in which the celebrity or deceased makes a relieved voice, and the image recognition / synthesis means generates an image in which the celebrity or deceased looks relieved.

[0658] The process flow of each embodiment will be described below.

[0659] "Example 1"

[0660] Step 1: The user boots the system.

[0661] Step 2: The system collects the user's personal data.

[0662] Step 3: The emotion engine recognizes the user's emotion.

[0663] Step 4: The generative AI generates a conversation based on the user's emotions.

[0664] Step 5: The speech recognition / synthesis means outputs the generated conversation as audio.

[0665] Step 6: The image recognition / synthesis means outputs an image corresponding to the generated conversation.

[0666] "Example 2"

[0667] Step 1: The user boots the system.

[0668] Step 2: The system collects the user's personal data.

[0669] Step 3: The emotion engine recognizes the user's emotion.

[0670] Step 4: The voice recognition / synthesis means and the image recognition / synthesis means generate voice and images according to the user's emotions.

[0671] Step 5: The generated audio and images are presented to the user.

[0672] Example 1

[0673] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0674] In systems that allow ordinary consumers to fulfill their wish to talk to celebrities or deceased people, conventional technology has had difficulty generating conversations that correspond to the user's emotions, and has been unable to provide a more natural and emotionally sensitive dialogue experience. In addition, real-time processing is difficult when generating voice and images, limiting the user experience.

[0675] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0676] In this invention, the server includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and an emotion-based conversation generation means. This enables natural conversation generation according to the user's emotion, and provides a richer conversation experience through real-time voice and image generation.

[0677] "Personal data collection means" refers to means for collecting data related to specific individuals, such as publicly available information and content posted on social networking services.

[0678] "Generative AI" is an artificial intelligence technology for generating natural conversations based on collected data.

[0679] The "voice recognition / synthesis means" refers to a means including a voice recognition technology that converts voice data into text data and a voice synthesis technology that converts text data into voice data.

[0680] "Image recognition / synthesis means" refers to means including image recognition technology that analyzes image data and extracts specific information, and image synthesis technology that generates a new image.

[0681] The "emotion recognition means" is a means for analyzing and recognizing the user's emotions from the user's voice, facial expressions, etc.

[0682] The "means for generating conversation according to emotions" refers to a means for generating appropriate conversation content based on the recognized emotions of the user.

[0683] This invention is a system for fulfilling the desire of ordinary consumers to converse with celebrities or the deceased, and includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and an emotion-based conversation generation means.

[0684] Server Operation

[0685] The server first uses the personal data collection means to collect personal data such as public information of celebrities and deceased people and posts on social networking services. This collection is carried out using a web crawler or an API. For example, the server uses the API to collect past posts of celebrities.

[0686] The server then pre-processes the collected personal data, filtering out unnecessary information and cleaning up the text data, removing advertisements and spam from collected posts, and normalizing the text using natural language processing techniques.

[0687] The preprocessed data is then input into a generative AI system to generate conversations that mimic the characteristics of celebrities and deceased individuals. A common natural language generation technique is used as the generative AI model for this generation. For example, the server inputs a prompt sentence such as "How was your day today?" and generates a response from the celebrity that reads, "Today was a very good day. My new project was a success."

[0688] The generated speech is then converted to audio using speech recognition / synthesis means, such as using text-to-speech software. For example, the server may send the generated text to a text-to-speech API to generate an audio file.

[0689] Additionally, the server may use image recognition / synthesis means to generate an image of the celebrity or deceased person, using image generation technology, for example, the server may use image generation technology to generate a facial image of the celebrity and create an image file for display to the user.

[0690] Using the emotion recognition means, the server recognizes the user's emotions. It sends the user's voice and facial expression data to the emotion recognition API and judges whether the user is happy or sad.

[0691] Finally, the server uses generative artificial intelligence to generate appropriate conversations based on the user's recognized emotions. For example, if the user is sad, the server generates comforting conversations such as, "It's okay, you're not alone. I've been through similar experiences."

[0692] Terminal operation

[0693] The device receives voice and text input from the user and sends it to the server. For example, if a user says to a smartphone, "How was your day today?", the voice data is sent to the server.

[0694] When the device receives the generated conversation, image, and audio data from the server, it displays and plays it for the user. For example, the device displays the generated image of a celebrity on the screen and plays the audio from a speaker.

[0695] User Actions

[0696] Users can access the system through a terminal and enjoy the experience of conversing with celebrities or deceased people. For example, if a user starts the app on their smartphone and speaks to the terminal, asking "How was your day today?", the user can experience a dialogue with a celebrity or deceased person through the conversation, images, and audio returned from the terminal.

[0697] Examples

[0698] The user speaks to the device, saying, "How was your day today?" The device sends the speech to the server. The server uses generative artificial intelligence to generate a conversation in which the celebrity or deceased person says something like, "It was a great day today. My new project was a success." The generated conversation is converted to voice using speech recognition / synthesis means and sent to the device. The device plays the voice to the user.

[0699] Examples of prompt statements

[0700] "Generate how a celebrity would respond if a user asked, 'How was your day?'"

[0701] In this way, users can experience interactions with celebrities and deceased people.

[0702] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0703] Step 1:

[0704] The server uses the personal data collection means to collect personal data such as public information of celebrities and deceased people and content posted on social networking services. Specifically, the server acquires data using a web crawler or an API. For example, the server uses an API to collect past posts of celebrities. The input is the URL or API endpoint to be collected, and the output is the collected text data.

[0705] Step 2:

[0706] The server preprocesses the collected personal data. It filters unnecessary information and cleans the text data. Specifically, it removes advertisements and spam from the collected posts and normalizes the text using natural language processing techniques. The input is the collected text data, and the output is the preprocessed clean text data.

[0707] Step 3:

[0708] The server inputs the preprocessed data into a generative AI system to generate a conversation that imitates the characteristics of celebrities or deceased people. A common natural language generation technique is used as the generative AI model. For example, the server inputs a prompt sentence such as "How was your day today?" and generates a response from the celebrity such as "Today was a very good day. My new project was successful." The input is the prompt sentence and the preprocessed text data, and the output is the generated conversation text.

[0709] Step 4:

[0710] The server converts the generated dialogue into audio using speech recognition / synthesis means. For this conversion, the server uses voice synthesis software. For example, the server sends the generated text to a speech synthesis API to generate an audio file. The input is the generated dialogue text and the output is the audio file.

[0711] Step 5:

[0712] The server uses image recognition / synthesis means to generate images of celebrities and deceased individuals. The image generation uses image generation technology. For example, the server uses image generation technology to generate facial images of celebrities and create image files to display to users. The input is the generated conversation text and related image data, and the output is the generated image file.

[0713] Step 6:

[0714] The server recognizes the user's emotions using emotion recognition means. It sends the user's voice and facial expression data to the emotion recognition API and judges whether the user is happy or sad. The input is the user's voice data and facial expression images, and the output is the recognized emotion data.

[0715] Step 7:

[0716] The server uses generative artificial intelligence to generate appropriate conversations according to the user's recognized emotions. For example, if the user is sad, the server generates a comforting conversation such as, "Don't worry, you're not alone. I've been through similar experiences." The input is the recognized emotion data and a prompt sentence, and the output is a conversation text according to the emotion.

[0717] Step 8:

[0718] The terminal receives voice and text input from the user and sends it to the server. For example, when a user speaks to a smartphone, "How was your day today?", the voice data is sent to the server. The input is the user's voice and text input, and the output is the transmission of data to the server.

[0719] Step 9:

[0720] The terminal receives the generated conversation, image, and audio data from the server. For example, the terminal displays the generated image of a celebrity on the screen and plays the audio from a speaker. The input is the data from the server, and the output is the display and playback to the user.

[0721] Step 10:

[0722] Users access the system through a terminal and enjoy the experience of conversing with celebrities or deceased people. For example, if a user starts the app on their smartphone and speaks to the terminal, asking "How was your day today?", the user can experience a dialogue with a celebrity or deceased person through the conversation, images, and audio returned from the terminal. The input is the user's operation, and the output is the dialogue experience.

[0723] (Application example 1)

[0724] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0725] Although there are systems that allow ordinary consumers to fulfill their desire to converse with celebrities or deceased people, these systems have the problem that they cannot provide dialogue that responds to the user's emotions or an interactive experience in a virtual store. In addition, there is a lack of technology that can track the user's location and movement to enable dialogue with celebrities or deceased people in real time. This may limit the user experience and reduce satisfaction.

[0726] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0727] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion engine, an interactive display means for interacting with celebrities and deceased people in a virtual store, and a location recognition means for tracking the user's location and movement. This enables conversations according to the user's emotions and interactive experiences in the virtual store, improving the user experience and satisfaction.

[0728] "Means of collecting personal data" refers to means of collecting personal data such as publicly available information about celebrities or the deceased, or the contents of social media posts.

[0729] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of celebrities or deceased people based on collected personal data.

[0730] "Speech recognition / synthesis means" refers to a means for recognizing a user's utterances and converting the generated conversation into voice.

[0731] "Image recognition / synthesis means" means means for generating images of celebrities or deceased persons and tracking the user's location and movements.

[0732] An "emotion engine" is a means by which a generative AI recognizes the user's emotions and generates conversations based on those emotions.

[0733] An "interactive display means for interacting with celebrities or deceased people in a virtual store" is a means for displaying celebrities or deceased people in a virtual store and interactively interacting with the user.

[0734] "Location awareness means for tracking a user's location and movements" means means for recognizing a user's location and movements in real time and providing an interactive experience.

[0735] The system for implementing this invention is configured using the following hardware and software. The hardware used is a smartphone and a head-mounted display (HMD). The software used is Unity (game engine), TensorFlow (generative AI model), Google Cloud Speech-to-Text (voice recognition), Google Cloud Text-to-Speech (voice synthesis), and OpenCV (image recognition).

[0736] The server includes personal data collection means, conversation generation means using generative AI, voice recognition / synthesis means, image recognition / synthesis means, an emotion engine, interactive display means for interacting with celebrities and deceased people in the virtual store, and location recognition means for tracking the user's location and movements.

[0737] First, the personal data collection means collects personal data such as public information of celebrities and deceased people and their social media posts. The collected data is analyzed using natural language processing (NLP) technology to extract important keywords and phrases.

[0738] Next, a generative AI-based conversation generator generates conversations that mimic the characteristics of celebrities or deceased people based on the collected personal data. The emotion engine adjusts the conversations to the user's emotions. For example, if the user is feeling happy, the generative AI generates conversations in which the celebrity or deceased person shares their joy.

[0739] The speech recognition / synthesis method uses Google Cloud Speech-to-Text to convert user speech into text in real time, and then converts the generated conversation into audio using Google Cloud Text-to-Speech.

[0740] The image recognition / synthesis method uses OpenCV to track the user's position and movements, generate 3D models of celebrities and deceased people within Unity, and display them interactively in response to the user's movements.

[0741] As a concrete example, when a user approaches the music section of a virtual store, a deceased famous musician appears and begins a conversation about the musician's life and music. When the user asks, "What is your favorite song?", the generative AI responds, "My favorite song is 'XX'," based on information obtained from the musician's interviews and social media posts.

[0742] An example of a prompt sentence is as follows:

[0743] User: What is your favorite song?

[0744] Generative AI model: My favorite song is "XX". This song has a special meaning in my life.

[0745] In this way, users can enjoy interacting with celebrities and deceased people within the virtual store.

[0746] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0747] Step 1:

[0748] The server uses the personal data collection means to collect personal data such as publicly available information and social media posts about celebrities and deceased people. As input, a web scraping tool (e.g., BeautifulSoup) is used to analyze the collected data using natural language processing (NLP) techniques to extract important keywords and phrases. As output, the analyzed personal data is obtained.

[0749] Step 2:

[0750] The server uses a generative AI-based conversation generation means to generate a conversation that mimics the characteristics of celebrities and deceased people based on the personal data collected in step 1. As input, the analyzed personal data is used to generate a conversation with a generative AI model (e.g., TensorFlow). As output, the generated conversation text is obtained.

[0751] Step 3:

[0752] The server uses an emotion engine to recognize the user's emotion and adjust the conversation generated in step 2 according to the user's emotion. As input, the server uses the user's emotion data and the generated conversation text to adjust the conversation in the emotion engine. As output, the adjusted conversation text is obtained.

[0753] Step 4:

[0754] The device uses speech recognition / synthesis to convert the user's speech into text in real time. As input, it uses the user's voice data and converts it into text using Google Cloud Speech-to-Text. As output, it obtains the text of the user's speech.

[0755] Step 5:

[0756] The server synthesizes the adjusted conversation text from step 3. As input, it uses the adjusted conversation text to generate speech data using Google Cloud Text-to-Speech. As output, it obtains the generated speech data.

[0757] Step 6:

[0758] The device tracks the user's location and movement using image recognition / synthesis. As input, it uses camera data and recognizes the user's location and movement using OpenCV. As output, it obtains the user's location data.

[0759] Step 7:

[0760] The server uses Unity to generate 3D models of celebrities and deceased people based on the user's location data obtained in step 6, and displays them interactively in the virtual store. The server uses the user's location data and the 3D model data of celebrities and deceased people as inputs, and generates interactive displays in Unity. The output is an interactive display in the virtual store.

[0761] Step 8:

[0762] Users enjoy interacting with celebrities and deceased people in a virtual store. As input, the system uses the user's speech and movements to generate dialogue and interact with the person through interactive displays. As output, the user's satisfaction is improved.

[0763] Example 2

[0764] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0765] In a system that allows ordinary consumers to fulfill their wish to talk to celebrities or deceased people, conventional technology had the problem that it was not possible to generate conversations or synthesize voice and images in real time, and it was not possible to respond according to the user's emotions. In addition, there was also the problem that the generated conversations, voices, and images were not natural, limiting the user experience.

[0766] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0767] In this invention, the server includes a personal data collection means, a conversation generation means using a generative AI model, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and an emotion-based voice and image generation means, which enables real-time conversation generation according to the user's emotions and the provision of natural voice and images.

[0768] "Personal data collection means" refers to a means for converting personal data provided by a user, such as letters, diaries, testimonies, etc., into digital format and storing it on a server.

[0769] A "generative AI model" is an artificial intelligence model that generates conversations that mimic the characteristics of a specific person based on collected personal data.

[0770] A "conversation generation means" is a means for generating a conversation based on collected personal data using a generative AI model.

[0771] The "voice recognition / synthesis means" refers to a means for converting the generated conversation data into voice and generating a voice that resembles the voice of a specific person.

[0772] An "image recognition / synthesis means" is a means for generating a 3D model from photographic data of a specific person and generating an image of that person.

[0773] The "emotion recognition means" is a means for analyzing the user's voice and facial expressions and recognizing the user's emotions.

[0774] The "means for generating voice and images according to emotions" refers to means for generating appropriate voice and images according to the user's emotions recognized by the emotion recognition means.

[0775] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0776] This invention is a system for enabling ordinary consumers to fulfill their desire to converse with celebrities or deceased people. This system includes a personal data collection means, a conversation generation means using a generative AI model, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and an emotion-based voice and image generation means.

[0777] Personal data collection methods

[0778] Users collect personal data such as past letters from grandparents or celebrities, diaries, and testimonials from family members. These data are converted into digital format using hardware such as scanners and cameras and stored on the terminal. The terminal then uploads the stored digital data to the server.

[0779] Conversation generation using generative AI models

[0780] The server receives the uploaded personal data and stores it in a database. It then uses a generative AI model (e.g., a general-purpose natural language processing model) to generate a conversation based on the personal data. This generative AI model analyzes the collected data and generates a natural conversation that mimics the characteristics of a specific person.

[0781] Speech recognition / synthesis means

[0782] The server sends the generated conversation data to a voice recognition / synthesis means (e.g., general-purpose voice synthesis software). The voice recognition / synthesis means converts the conversation data into a voice that resembles the voice of a specific person. The generated voice data is temporarily stored on the server.

[0783] Image recognition / synthesis methods

[0784] The server sends photo data of a specific person to an image recognition / synthesis means (e.g., general-purpose image generation technology). The image recognition / synthesis means generates a 3D model from the photo data and creates an image of the person. The generated image data is temporarily stored on the server.

[0785] emotion recognition means

[0786] While the user is using the system, the server transmits the user's voice and facial expression in real time to an emotion recognition means (e.g., general-purpose emotion analysis software). The emotion recognition means analyzes the user's emotions and sends feedback to the server.

[0787] Means for generating sounds and images according to emotions

[0788] The server issues instructions to the voice recognition / synthesis means and the image recognition / synthesis means based on feedback from the emotion recognition means to generate appropriate voice and image. For example, if the user feels surprised, the voice recognition / synthesis means generates a surprised voice, and the image recognition / synthesis means generates a surprised image.

[0789] Examples

[0790] Example 1: Conversation with grandparents

[0791] If a user wants to experience a conversation with their grandparents, they can input the following prompt into the generative AI model:

[0792] "Grandpa, please tell me some of your memories from the past."

[0793] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the grandparents' characteristics. It then uses a speech recognition / synthesis means to generate a voice that resembles the grandparents' voice, and an image recognition / synthesis means to generate an image of the grandparents.

[0794] Example 2: Conversation with a celebrity

[0795] If a user wants to experience a conversation with a celebrity, they can input the following prompt into the generative AI model:

[0796] "Celebrities, tell us about your accomplishments."

[0797] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the celebrity's characteristics, then uses a speech recognition / synthesis method to generate a voice similar to the celebrity's voice, and uses an image recognition / synthesis method to generate an image of the celebrity.

[0798] The system allows users to experience realistic conversations with deceased people or celebrities.

[0799] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0800] System program processing flow

[0801] Step 1: Collecting personal data

[0802] Users collect personal data such as past letters from grandparents or celebrities, diaries, testimonials from family members, etc. This data is converted into digital form using hardware such as scanners and cameras and stored on the device.

[0803] Input: Physical data such as letters, diaries, testimonies, etc.

[0804] Output: Personal data in digital format

[0805] Step 2: Upload your data

[0806] The terminal uploads the stored digital data to a server.

[0807] Input: Personal data in digital form

[0808] Output: Digital data stored on a server

[0809] Step 3: Conversation generation

[0810] The server receives the uploaded personal data, stores it in a database, and then uses a generative AI model to generate a conversation based on the personal data.

[0811] Input: Personal data in digital form

[0812] Output: Generated conversation data

[0813] Step 4: Text-to-speech

[0814] The server transmits the generated conversation data to a voice recognition / synthesis means, which converts the conversation data into a voice that resembles the voice of a specific person.

[0815] Input: Generated conversation data

[0816] Output: Generated audio data

[0817] Step 5: Image Generation

[0818] The server transmits photographic data of a particular person to the image recognition / synthesis means, which generates a 3D model from the photographic data and creates an image of the person.

[0819] Input: Photo data of a specific person

[0820] Output: Generated image data

[0821] Step 6: Emotion Recognition

[0822] While the user is using the system, the server transmits the user's voice and facial expression in real time to the emotion recognition means, which analyzes the user's emotions and sends feedback to the server.

[0823] Input: User's voice and facial expression data

[0824] Output: Parsed emotion data

[0825] Step 7: Emotion-based voice and image generation

[0826] The server issues instructions to the voice recognition / synthesis means and the image recognition / synthesis means based on feedback from the emotion recognition means to generate appropriate voice and image. For example, if the user feels surprised, the voice recognition / synthesis means generates a surprised voice, and the image recognition / synthesis means generates a surprised image.

[0827] Input: Parsed emotion data

[0828] Output: Voice data and image data according to emotions

[0829] Examples

[0830] Example 1: Conversation with grandparents

[0831] If a user wants to experience a conversation with their grandparents, they can input the following prompt into the generative AI model:

[0832] "Grandpa, please tell me some of your memories from the past."

[0833] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the grandparents' characteristics. It then uses a speech recognition / synthesis means to generate a voice that resembles the grandparents' voice, and an image recognition / synthesis means to generate an image of the grandparents.

[0834] Example 2: Conversation with a celebrity

[0835] If a user wants to experience a conversation with a celebrity, they can input the following prompt into the generative AI model:

[0836] "Celebrities, tell us about your accomplishments."

[0837] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the celebrity's characteristics, then uses a speech recognition / synthesis method to generate a voice similar to the celebrity's voice, and uses an image recognition / synthesis method to generate an image of the celebrity.

[0838] The system allows users to experience realistic conversations with deceased people or celebrities.

[0839] (Application example 2)

[0840] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0841] Conventional technology has made it difficult for ordinary consumers to experience conversations with deceased people or celebrities. In particular, providing a realistic conversation experience that corresponds to the user's emotions is technically complex and difficult to achieve. Furthermore, there has been no system that can easily provide such an experience using a mobile terminal such as a smartphone. As a result, users have been unable to enjoy a realistic conversation experience that corresponds to their emotions.

[0842] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0843] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion engine, a means for generating voice and images according to the user's emotion, and an application installed on a smartphone, which allows the user to experience a real conversation with a deceased person or a celebrity.

[0844] "Personal data collection means" are means for collecting personal information of deceased persons or celebrities, such as past letters or diaries, or testimonies from family members.

[0845] A "conversation generation method using generative AI" is a method that uses artificial intelligence technology to generate conversations that imitate the characteristics of a deceased person or celebrity based on collected personal data.

[0846] The "voice recognition / synthesis means" refers to a means for recognizing the user's voice and converting the generated conversation into voice.

[0847] "Image recognition / synthesis means" means means for generating images using 3D models generated from photographs of deceased persons or celebrities.

[0848] An "emotion engine" is a means for recognizing a user's emotions and generating sounds and images according to those emotions.

[0849] "Means for generating voice and images according to the user's emotions" refers to means for controlling voice recognition / synthesis means and image recognition / synthesis means based on the emotions felt by the user, and generating appropriate voice and images.

[0850] The "application installed on a smartphone" is an application that runs on a smartphone and allows the user to enjoy the experience of having a conversation with a deceased person or a celebrity.

[0851] The system for implementing this invention uses the following hardware and software. The hardware requires a smartphone, a microphone, and a speaker. The software uses Python, OpenAI API, Google Text-to-Speech (gTTS), OpenCV, and SpeechRecognition.

[0852] The server uses a personal data collection means to collect personal information of the deceased or famous person, such as past letters and diaries, testimonials from family members, etc. This collected data is analyzed by a conversation generation means using generative AI, and conversations that imitate the characteristics of the deceased or famous person are generated.

[0853] The terminal (smartphone) uses a voice recognition / synthesis means to recognize the user's voice and convert the generated conversation into audio.Furthermore, it uses an image recognition / synthesis means to generate an image using a 3D model generated from a photograph of a deceased person or a celebrity.

[0854] The emotion engine recognizes the user's emotion and generates voice and image according to the emotion. For example, if the user feels surprised, the voice recognition / synthesis means generates a voice that sounds surprised, and the image recognition / synthesis means generates an image that looks surprised. On the other hand, if the user feels relieved, the voice recognition / synthesis means generates a voice that sounds relieved, and the image recognition / synthesis means generates an image that looks relieved.

[0855] As a concrete example, if a user says, "What happened today?", the emotion engine recognizes the user's emotion as "surprise." In this case, the generative AI generates a conversation using the following prompt sentence:

[0856] Example prompt:

[0857] “Generate conversations when users have an emotion of surprise: What happened today?”

[0858] By inputting this prompt sentence into the generative AI model, a conversation corresponding to the emotion of surprise is generated. The generated conversation is converted into voice by the speech recognition / synthesis means, and an image with a surprised facial expression is generated by the image recognition / synthesis means.

[0859] As described above, the present invention is a system that enables a user to experience a realistic conversation with a deceased person or a celebrity.

[0860] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0861] Step 1:

[0862] The server uses the personal data collection means to collect personal information such as past letters and diaries of deceased people and celebrities, and testimonials from family members. The input is personal data such as letters, diaries, and testimonials, and the output is a database of the collected personal data. Specifically, the server saves the data in the database and organizes and classifies the data as necessary.

[0863] Step 2:

[0864] The server analyzes the collected personal data using a generative AI-based conversation generation means to generate conversations that mimic the characteristics of deceased people or celebrities. The input is the collected personal data, and the output is the generated conversation text. Specifically, the server inputs a prompt sentence into the generative AI model to generate the conversation text.

[0865] Step 3:

[0866] The terminal (smartphone) recognizes the user's voice using a speech recognition / synthesis means. The input is the user's voice, and the output is the recognized text. Specifically, the voice is captured using a microphone and converted to text using the SpeechRecognition library.

[0867] Step 4:

[0868] The terminal uses an emotion engine to recognize the user's emotion. The input is the recognized text, and the output is the recognized emotion. Specifically, the terminal performs text analysis and estimates the emotion.

[0869] Step 5:

[0870] The server uses generative AI to generate a conversation according to the user's emotions. The input is the recognized emotion and the user's text, and the output is a conversation text according to the emotion. Specifically, it generates a prompt sentence and inputs it into the generative AI model to generate the conversation text. An example of a prompt sentence is "Generate a conversation when the user has an emotion of surprise: What happened today?"

[0871] Step 6:

[0872] The terminal uses a speech recognition / synthesis means to convert the generated conversation text into speech. The input is the generated conversation text, and the output is an audio file. Specifically, the terminal uses Google Text-to-Speech (gTTS) to convert the text into speech and generate an audio file.

[0873] Step 7:

[0874] The device uses image recognition / synthesis to generate images using 3D models generated from photos of deceased people or celebrities. The input is the recognized emotion and the 3D model, and the output is an image corresponding to the emotion. Specifically, the device uses OpenCV to generate an image with a facial expression corresponding to the emotion.

[0875] Step 8:

[0876] The terminal provides the generated audio and images to the user. The input is an audio file and an image, and the output is the audio and images provided to the user. In concrete terms, the terminal operates by playing the audio using a speaker and displaying the images using a display.

[0877] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0878] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0879] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0880] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0881] [Third embodiment]

[0882] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0883] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0884] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0885] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0886] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0887] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0888] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0889] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0890] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0891] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0892] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0893] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0894] "Example 1"

[0895] As one embodiment of the present invention, a system is provided that allows general consumers to experience conversations with celebrities and deceased persons. This system is configured by combining a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, and an image recognition / synthesis means. The personal data collection means collects personal data such as public information of celebrities and deceased persons and SNS posts. The generative AI generates a conversation that imitates the characteristics of the celebrity or deceased person based on the collected personal data. The voice recognition / synthesis means converts the generated conversation into voice. The image recognition / synthesis means generates an image of the celebrity or deceased person. By combining these means, a system is realized that allows general consumers to experience conversations with celebrities and deceased persons.

[0896] "Example 2"

[0897] As a specific example, consider a system that allows general consumers to experience a conversation with their deceased grandparents. First, a personal data collection means collects personal data such as the grandparents' past letters and diaries, and testimonies from family members. A generative AI generates conversation that mimics the characteristics of the grandparents based on the collected personal data. A voice recognition / synthesis means synthesizes the generated conversation into a voice that is close to the grandparents' voice. An image recognition / synthesis means generates images of the grandparents using 3D models generated from photos of the grandparents. By combining these means, a system is realized that allows general consumers to experience a conversation with their deceased grandparents.

[0898] The process flow of each embodiment will be described below.

[0899] "Example 1"

[0900] Step 1: Personal data such as publicly available information and social media posts about celebrities and deceased people is collected using personal data collection methods.

[0901] Step 2: Generative AI generates conversations that mimic the characteristics of celebrities or deceased people based on the collected personal data.

[0902] Step 3: The speech recognition / synthesis means converts the generated speech into audio.

[0903] Step 4: Image recognition / synthesis procedures generate an image of the celebrity or deceased person.

[0904] "Example 2"

[0905] Step 1: Personal data collection instruments are used to collect personal data such as grandparents' past letters and diaries, and testimonies from family members.

[0906] Step 2: The generative AI generates conversations that mimic the characteristics of the grandparents based on the collected personal data.

[0907] Step 3: The speech recognition / synthesis means synthesizes the generated conversation into a voice that resembles the grandparents' voice.

[0908] Step 4: The image recognition / synthesis means generates images of the grandparents using the 3D models generated from the photos of the grandparents.

[0909] Example 1

[0910] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0911] There is a demand for a more realistic and natural conversation experience in a system that allows ordinary consumers to fulfill their wish to talk to celebrities or deceased people. However, conventional systems have limitations in the quality of collected data, the naturalness of generated conversations, and the realism of voice and images, resulting in an insufficient user experience.

[0912] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0913] In this invention, the server includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition means, a voice synthesis means, an image recognition means, and an image generation means, which enable the server to generate a conversation that imitates the characteristics of a specific person based on the collected personal data, convert a user's voice input into text, vocalize the generated conversation, and recognize and generate an image of the specific person.

[0914] "Personal data collection means" refers to means for collecting data such as publicly available information about a specific person and the contents of social media posts.

[0915] "Generative AI" is AI that generates conversations that mimic the characteristics of a specific person based on collected data.

[0916] A "speech recognition means" is a means for converting a user's voice input into text.

[0917] The "voice synthesis means" is a means for converting the generated conversation into voice.

[0918] The "image recognition means" is a means for recognizing an image of a specific person.

[0919] An "image generating means" is a means for generating an image of a particular person.

[0920] This invention is a system for fulfilling the desire of ordinary consumers to converse with celebrities or deceased people, and includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition means, a voice synthesis means, an image recognition means, and an image generation means.

[0921] Personal data collection methods

[0922] The server collects personal data such as public information and social media posts of celebrities and deceased people. This data is collected using web scraping tools (e.g., Beautiful Soup and Scrapy) and social media APIs (e.g., Twitter API and Facebook Graph API). The collected data is stored in a database.

[0923] Conversation Generation Method

[0924] The server generates a conversation using generative artificial intelligence (e.g., OpenAI's GPT-4) based on the collected personal data. The generative artificial intelligence analyzes the input personal data and generates natural conversational sentences. An example of a prompt sentence is, "Based on the public information and SNS posts of famous singer A, please generate a conversation that imitates his characteristics."

[0925] Speech recognition and synthesis means

[0926] The server converts the generated conversational text into voice using speech synthesis software (e.g., Google Text-to-Speech or Amazon Polly). It also converts the user's voice input into text using speech recognition software (e.g., Google Speech-to-Text or IBM Watson Speech to Text). This allows the user's question to be obtained as text and input into the generative artificial intelligence.

[0927] Image recognition and image generation

[0928] The server uses image generation software (e.g. DALL-E or DeepArt) to generate images of celebrities or deceased people, allowing the user to visually recognize the person they are speaking with. It inputs prompts to the image generation software based on the collected personal data, and sends the generated image to the user's device.

[0929] User interaction

[0930] Users access the system through a terminal (e.g., a smartphone or PC) and experience conversations with celebrities or deceased people. When the user asks a question by voice, the voice recognition means converts the question into text, and the generative artificial intelligence generates an answer. The generated answer is converted into voice by the voice synthesis means and returned to the user. At the same time, an image of the celebrity or deceased person, generated by the image recognition means and the image generation means, is displayed.

[0931] Examples

[0932] For example, if a user wishes to "talk to famous singer A," the system operates as follows.

[0933] 1. The server collects public information and social media posts about famous singer A.

[0934] 2. The server uses generative artificial intelligence based on the collected data to generate conversations that mimic the characteristics of famous singer A.

[0935] 3. The server converts the generated conversational text into voice using speech synthesis software.

[0936] 4. The server generates an image of famous singer A.

[0937] 5. The user experiences a conversation with famous singer A through the terminal using audio and images.

[0938] In this way, a system is realized that allows ordinary consumers to experience conversations with celebrities and deceased people.

[0939] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0940] Step 1:

[0941] The server uses personal data collection means to collect public information and social media posts about celebrities and deceased people. Specifically, it uses web scraping tools (e.g. Beautiful Soup and Scrapy) and social media APIs (e.g. Twitter API and Facebook Graph API). The input is the name and related information of a specific person, and the output is the collected personal data. The collected data is stored in a database.

[0942] Step 2:

[0943] The server retrieves the collected personal data from the database and generates a conversation using generative artificial intelligence (e.g., OpenAI's GPT-4). The input is the collected personal data, and the output is the generated conversation. Specifically, a prompt sentence is input into the generative artificial intelligence, and a conversation sentence that imitates the characteristics of a specific person is generated. An example of a prompt sentence is, "Based on the public information of famous singer A and his social media posts, please generate a conversation that imitates his characteristics."

[0944] Step 3:

[0945] The server converts the generated conversation text into voice using speech synthesis software (e.g., Google Text-to-Speech or Amazon Polly). The input is the generated conversation text, and the output is an audio file. Specifically, the generated conversation text is input into the speech synthesis software to generate an audio file.

[0946] Step 4:

[0947] The server converts the user's voice input into text using voice recognition software (e.g., Google Speech-to-Text or IBM Watson Speech to Text). The input is the user's voice, and the output is text. Specifically, the user's voice input is input into the voice recognition software and converted into text.

[0948] Step 5:

[0949] The server generates an image of a specific person using image generation software (e.g., DALL-E or DeepArt) based on the collected personal data. The input is the collected personal data, and the output is the generated image. Specifically, a prompt is input to the image generation software, and an image of a specific person is generated.

[0950] Step 6:

[0951] Users access the system through a terminal (e.g., a smartphone or PC) and experience a conversation with a celebrity or a deceased person. The input is the user's voice question, and the output is the generated audio and image of the conversation. Specifically, when the user asks a question by voice, the voice recognition means converts the question into text, and the generative artificial intelligence generates an answer. The generated answer is converted into voice by the voice synthesis means and returned to the user. At the same time, an image of the celebrity or deceased person, generated by the image recognition means and the image generation means, is displayed.

[0952] (Application example 1)

[0953] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0954] When ordinary consumers wish to converse with celebrities or deceased people, it is difficult to provide a realistic experience that would allow them to realize their wish. In particular, advanced technology is required to provide a real-time conversation experience that combines voice and images. There is also a demand to provide this experience in a form that is easy for users to use.

[0955] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0956] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, an application means installed on a smartphone, a means for recognizing a user's voice input and generating a voice response in real time, and a means for generating an image of a celebrity or a deceased person and displaying it during a conversation. This allows general consumers to easily enjoy a real conversation experience with a celebrity or a deceased person.

[0957] "Personal data collection means" refers to means for collecting personal data, such as publicly available information about celebrities or the deceased, or the content of social media posts.

[0958] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of celebrities or deceased people based on collected personal data.

[0959] "Speech recognition / synthesis means" means for recognizing a user's voice input and converting the generated speech into voice.

[0960] "Image recognition / synthesis means" refers to means for generating images of celebrities or deceased people and displaying them during a conversation.

[0961] The "application means installed on a smartphone" refers to a means that is installed on a smartphone and provides an application that enables a user to have a conversation experience with a celebrity or a deceased person.

[0962] The "means for recognizing a user's voice input and generating a voice response in real time" refers to a means for recognizing a user's voice input in real time and generating a voice response in response thereto.

[0963] "Means for generating an image of a celebrity or a deceased person and displaying it during a conversation" refers to means for generating an image of a celebrity or a deceased person and displaying it to a user during a conversation.

[0964] The system for carrying out the present invention is configured as follows. First, the server uses a personal data collection means to collect personal data such as public information and social media posts of celebrities and deceased people. This collected data is input to a conversation generation means using generative AI, and conversations that imitate the characteristics of celebrities and deceased people are generated.

[0965] Next, an application installed on the user's device (smartphone) uses speech recognition / synthesis means to recognize the user's voice input and generates a real-time voice response to convert the generated conversation into voice. It also uses image recognition / synthesis means to generate images of celebrities and deceased people and display them to the user during the conversation.

[0966] Specifically, the server uses the following software and hardware: It uses the OpenAI API for generative AI, the SpeechRecognition library for voice recognition, and the gTTS library for voice synthesis. It uses the OpenCV library for image processing. These software run on the server and work in conjunction with applications installed on the user's smartphone.

[0967] For example, if a user wants to talk to a "historical figure," the server collects personal data about that person and uses generative AI to generate a conversation that mimics the characteristics of that person. When the user asks a question to the smartphone, the voice recognition / synthesis means recognizes the question and generates a voice response in real time. At the same time, the image recognition / synthesis means generates an image of the person and displays it on the smartphone screen.

[0968] An example of a prompt is as follows:

[0969] "Collect personal data about a historical figure and generate answers to the following user questions that mimic the characteristics of that person.

[0970] User: What is your dream?

[0971] AI:

[0972] In this way, users can easily enjoy the experience of having a real conversation with a celebrity or a deceased person.

[0973] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0974] Step 1:

[0975] The server uses the personal data collection means to collect personal data such as public information and social media posts of celebrities and deceased individuals. The input is the name of the celebrity or deceased individual, and the output is the collected personal data. This data collection is done through public information on the Internet and social media APIs.

[0976] Step 2:

[0977] The server inputs the collected personal data into a generative AI to generate a conversation that imitates the characteristics of celebrities and deceased people. The input is the collected personal data, and the output is the generated conversation text. The generative AI uses OpenAI's API to generate a conversation based on the prompt sentence.

[0978] Step 3:

[0979] The user's device (smartphone) recognizes the user's voice input using a speech recognition / synthesis means. The input is the user's voice and the output is the user's question in text form. The SpeechRecognition library is used for speech recognition.

[0980] Step 4:

[0981] The server inputs the user's question text into the generative AI, which generates a voice response in real time. The input is the user's question text, and the output is the generated response text. The generative AI again uses OpenAI's API to generate a response based on the prompt text.

[0982] Step 5:

[0983] The user's terminal inputs the generated response text to a speech synthesis means and converts it into speech. The input is the generated response text, and the output is an audio file. The gTTS library is used for speech synthesis.

[0984] Step 6:

[0985] The server generates images of celebrities and deceased people using image recognition / synthesis methods. The input is the name of the celebrity or deceased person, and the output is the generated image file. The OpenCV library is used for image generation.

[0986] Step 7:

[0987] The user's terminal displays the generated audio and image files. The input is the audio and image files, and the output is visual and auditory feedback to the user. The audio files are played through the speaker, and the image files are shown on the display.

[0988] Example 2

[0989] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0990] Conventional technology has made it difficult for ordinary consumers to experience conversations with deceased people or specific individuals. In particular, generating natural conversations that reflect the characteristics of the deceased and realistic voices and images is technically complex and difficult to achieve. For this reason, there is a demand for a system that can reproduce conversations with the deceased.

[0991] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0992] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a means for providing a conversation experience by combining the generated voice and image, thereby enabling general consumers to experience a natural conversation with a deceased person or a specific person.

[0993] "Personal data collection means" refers to devices or methods for collecting and digitizing personal information of a specific individual, such as past letters, diaries, and testimonies.

[0994] A "conversation generation method using generative AI" is an artificial intelligence technology for generating conversations that mimic the characteristics of a specific individual based on collected personal data.

[0995] "Speech recognition / synthesis means" refers to technology for converting the generated conversation into voice data and generating a voice that resembles the voice of a specific person.

[0996] "Image recognition / synthesis means" is a technology for generating 3D models and realistic images based on photographs or videos of specific people.

[0997] "Means for combining generated audio and images to provide a conversational experience" refers to an apparatus or method for synchronizing generated audio data and image data to provide a user with an interactive conversational experience.

[0998] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0999] This invention is a system for enabling general consumers to experience conversations with deceased people or specific individuals. The system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a means for providing a conversation experience by combining the generated voice and image.

[1000] Collection of Personal Data

[1001] Users collect personal data such as past letters from grandparents or specific individuals, diaries, and testimonials from family members. These data are digitized using a scanner, voice recording device, and text input software. For example, a user can use a scanner to scan letters from grandparents and save them as text data.

[1002] Conversation generation using generative AI

[1003] The server receives the collected personal data and uses a generative AI model (e.g., a general natural language processing model) to generate a conversation that mimics the characteristics of grandparents or a specific person. Specifically, the server analyzes the data and inputs a prompt sentence into the generative AI model. For example, the server inputs a prompt sentence such as, "Based on the contents of the letter from my grandparents, please generate a conversation that reflects the characteristics of my grandparents."

[1004] Speech Recognition / Synthesis

[1005] The server uses a voice recognition / synthesis means (for example, a general voice synthesis API) to synthesize the generated conversation into a voice that resembles the voice of a grandparent or a specific person. Specifically, the server sends the generated text data to a voice synthesis API and obtains a voice file. For example, the generated conversation, "It's a very nice day today," is synthesized into the voice of a grandparent.

[1006] Image Recognition / Synthesis

[1007] The server uses image recognition / synthesis means (e.g., a general image generation algorithm) to generate a 3D model using photos of the grandparents or a specific person. Specifically, the server analyzes the photo data and inputs it into a 3D model generation algorithm. For example, a 3D model is generated based on a photo of the grandparents' faces, and an avatar that moves in real time is created.

[1008] Providing a conversational experience

[1009] The device combines the generated voice and image to provide the user with a conversation experience with a grandparent or a specific person. Specifically, the device plays the audio file and the 3D model in sync, allowing the user to have an interactive conversation. For example, the user can talk to an avatar of a grandparent while looking at the screen of the device.

[1010] Examples and prompts

[1011] Examples

[1012] A user scans a letter from their grandparents and uploads it to the system. The server analyzes the contents of the letter and uses a generative AI model to generate a conversation that mimics the characteristics of the grandparents. The generated conversation is then converted into voice using a common voice synthesis API, and a 3D model of the grandparent is generated using a common image generation algorithm. Finally, the user can talk to the grandparents through the device.

[1013] Examples of prompt statements

[1014] "Based on the content of the letter below, please generate a conversation that reflects the characteristics of the grandparents. The content of the letter is as follows: 'Dear Grandson, It's a very nice day today and the flowers in the garden are in full bloom. If you were here, we could go for a walk together.'"

[1015] By inputting this prompt into the generative AI model, natural conversation that reflects the characteristics of the grandparents is generated.

[1016] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1017] Step 1:

[1018] The user collects personal data such as past letters, diaries, and testimonials from family members of grandparents or other specific individuals. These data are digitized using a scanner, a voice recording device, and text input software. Specifically, the user uses a scanner to scan letters from grandparents and saves them as text data. The input is physical personal data such as letters and diaries, and the output is digitized text data.

[1019] Step 2:

[1020] The server receives digitized personal data uploaded by the user. It then uses a generative AI model (e.g., a general natural language processing model) to generate a conversation that mimics the characteristics of a specific person. Specifically, the server analyzes the data and inputs a prompt sentence into the generative AI model. For example, the server inputs the prompt sentence, "Based on the contents of my grandparents' letters, please generate a conversation that reflects my grandparents' characteristics." The input is the digitized personal data, and the output is the generated conversation text.

[1021] Step 3:

[1022] The server synthesizes the generated conversation text into a voice that resembles the voice of a specific person using a voice recognition / synthesis means (for example, a general voice synthesis API). Specifically, the server sends the generated text data to a voice synthesis API and obtains an audio file. For example, the generated conversation "It's a very nice day today" is synthesized in the voice of a grandparent. The input is the generated conversation text, and the output is an audio file.

[1023] Step 4:

[1024] The server uses image recognition / synthesis means (e.g., a general image generation algorithm) to generate a 3D model using a photo of a specific person. Specifically, the server analyzes the photo data and inputs it into a 3D model generation algorithm. For example, a 3D model is generated based on a photo of a grandparent's face, and an avatar that moves in real time is created. The input is photo data of a specific person, and the output is a 3D model.

[1025] Step 5:

[1026] The terminal combines the generated audio file with the 3D model to provide the user with a conversation experience with a specific person. Specifically, the terminal plays the audio file and the 3D model in sync, allowing the user to have an interactive conversation. For example, a user can interact with an avatar of their grandparents while looking at the screen of the terminal. The input is the audio file and the 3D model, and the output is an interactive conversation experience.

[1027] (Application example 2)

[1028] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1029] There are existing systems that allow ordinary consumers to fulfill their desire to converse with deceased relatives or celebrities, but these systems have the problem of lacking realism and limiting the user experience. In addition, there is a lack of means to realize dialogue in a virtual space, making it difficult to provide a more realistic experience. This has led to a demand for a system that allows users to experience a more realistic conversation with deceased relatives or celebrities.

[1030] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1031] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a display means for realizing a conversation in a virtual space, which allows general consumers to experience a more realistic conversation with a deceased person or a celebrity.

[1032] "Personal data collection methods" are methods for collecting data about individuals, such as past letters, diaries, and testimonies from family members of deceased or famous people.

[1033] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of deceased people or celebrities based on collected personal data.

[1034] The "voice recognition / synthesis means" refers to a means for converting the generated conversation into voice and synthesizing a voice that is close to the voice of the deceased or famous person.

[1035] "Image recognition / synthesis means" refers to a means for generating 3D models from photographs or videos of deceased persons or celebrities and synthesizing realistic images.

[1036] "Display means for realizing dialogue in a virtual space" refers to a means for displaying generated conversations and images in a virtual space, providing a user with the experience of interacting with a deceased person or a celebrity.

[1037] The following system configuration will be described as an embodiment of the present invention.

[1038] System Configuration

[1039] This system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a display means for realizing dialogue in a virtual space.

[1040] Personal data collection methods

[1041] The server collects data about deceased or famous people, such as past letters, diaries, and testimonials from family members. This data may be provided by users or collected from public sources on the Internet.

[1042] Conversation generation using generative AI

[1043] Based on the collected personal data, the server uses generative AI to generate conversations that mimic the characteristics of deceased people and celebrities. Specifically, it uses OpenAI's API to generate conversations based on prompt sentences.

[1044] Speech recognition / synthesis means

[1045] The server uses a speech synthesis engine (e.g., pyttsx3) to convert the generated conversation into audio and synthesize a voice that is close to the voice of the deceased person or celebrity, allowing users to experience the conversation with a realistic voice.

[1046] Image recognition / synthesis methods

[1047] The server uses image processing libraries (e.g., OpenCV) to generate 3D models from photos and videos of deceased people and celebrities and synthesize realistic images, allowing users to experience realistic interactions.

[1048] Display means for realizing dialogue in virtual space

[1049] The device displays the generated conversations and images in a virtual space, allowing users to have the experience of interacting with deceased people or celebrities. Specifically, the device uses a smartphone and a head-mounted display to realize the interaction in the virtual space.

[1050] Examples

[1051] For example, consider the case where a user uploads a letter from their grandparents to an app and experiences the generated conversation. The user can talk to their grandparents in a virtual space using a smartphone or a head-mounted display. In this case, the following prompt sentence is used:

[1052] Examples of prompt statements

[1053] "Generate conversations by mimicking your grandparents' traits: Grandparents' letters, diaries, and family testimonials."

[1054] In this way, consumers can have a more realistic conversation experience with deceased people or celebrities.

[1055] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1056] Step 1:

[1057] Users use the device to upload personal data (letters, diaries, testimonials, etc.) of deceased people or celebrities.

[1058] Input: Personal data such as letters, diaries, testimonies, etc.

[1059] Output: Personal data stored on the server

[1060] Specific operation: A user uploads personal data to the application using a smartphone or PC. The uploaded data is stored on the server.

[1061] Step 2:

[1062] The server analyzes the personal data and sends a prompt to the generative AI.

[1063] Input: Personal data stored on the server

[1064] Output: The prompt sent to the generative AI

[1065] Specific operation: The server analyzes the uploaded personal data and generates a prompt to send to the generative AI. For example, it generates a prompt such as, "Please generate a conversation by imitating the characteristics of my grandparents: letters and diaries from my grandparents, and testimonials from family members."

[1066] Step 3:

[1067] A generative AI generates a conversation based on the prompt.

[1068] Input: Prompt sent to the generative AI

[1069] Output: Generated conversation text

[1070] Specific operation: The generative AI (e.g., OpenAI's API) generates a conversation text based on the prompt sentence. The generated conversation text is returned to the server.

[1071] Step 4:

[1072] The server transmits the generated conversation text to a speech recognition / synthesis means to generate speech.

[1073] Input: Generated conversation text

[1074] Output: Generated audio data

[1075] Specific operation: The server sends the generated conversation text to a speech synthesis engine (e.g., pyttsx3) to generate a voice similar to the voice of the deceased or famous person. The generated voice data is stored on the server.

[1076] Step 5:

[1077] The server sends photos of the deceased person or celebrity to an image recognition / synthesis tool, which generates a 3D model.

[1078] Input: A photo of a deceased or famous person

[1079] Output: The generated 3D model

[1080] How it works: The server sends photos of the deceased person or celebrity to an image processing library (e.g., OpenCV) to generate a 3D model, which is then stored on the server.

[1081] Step 6:

[1082] The device receives the generated voice data and 3D model and displays it in the virtual space.

[1083] Input: Generated voice data, generated 3D model

[1084] Output: Interactive experience in virtual space

[1085] Specific operation: The device (smartphone or head-mounted display) uses the voice data and 3D model received from the server to display a conversation with the deceased person or celebrity in a virtual space, allowing the user to have a realistic conversation experience.

[1086] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1087] "Example 1"

[1088] In one embodiment of the present invention, a system is provided for enabling general consumers to fulfill their desire to converse with celebrities or deceased people. The system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and an emotion engine. The emotion engine recognizes the user's emotions, and the generative AI generates a conversation in response to the emotions. Specifically, when the user is feeling joy, the generative AI generates a conversation in which the celebrity or the deceased person shares the joy. Also, when the user is feeling sad, the generative AI generates a conversation in which the celebrity or the deceased person comforts the user.

[1089] "Example 2"

[1090] In another embodiment of the present invention, a system is provided for enabling general consumers to fulfill their desire to converse with celebrities or deceased people. The system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and an emotion engine. The emotion engine recognizes the user's emotion, and the voice recognition / synthesis means and the image recognition / synthesis means generate voice and image in response to the emotion. Specifically, when the user feels surprised, the voice recognition / synthesis means generates a voice in which the celebrity or deceased makes a surprised voice, and the image recognition / synthesis means generates an image in which the celebrity or deceased looks surprised. Also, when the user feels relieved, the voice recognition / synthesis means generates a voice in which the celebrity or deceased makes a relieved voice, and the image recognition / synthesis means generates an image in which the celebrity or deceased looks relieved.

[1091] The process flow of each embodiment will be described below.

[1092] "Example 1"

[1093] Step 1: The user boots the system.

[1094] Step 2: The system collects the user's personal data.

[1095] Step 3: The emotion engine recognizes the user's emotion.

[1096] Step 4: The generative AI generates a conversation based on the user's emotions.

[1097] Step 5: The speech recognition / synthesis means outputs the generated conversation as audio.

[1098] Step 6: The image recognition / synthesis means outputs an image corresponding to the generated conversation.

[1099] "Example 2"

[1100] Step 1: The user boots the system.

[1101] Step 2: The system collects the user's personal data.

[1102] Step 3: The emotion engine recognizes the user's emotion.

[1103] Step 4: The voice recognition / synthesis means and the image recognition / synthesis means generate voice and images according to the user's emotions.

[1104] Step 5: The generated audio and images are presented to the user.

[1105] Example 1

[1106] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1107] In a system that allows ordinary consumers to fulfill their wish to talk to celebrities or deceased people, conventional technology has difficulty generating conversations according to the user's emotions, and has been unable to provide a more natural and emotionally sensitive dialogue experience. In addition, real-time processing is difficult when generating voice and images, limiting the user experience.

[1108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1109] In this invention, the server includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and a conversation generation means according to emotion. This enables natural conversation generation according to the user's emotion, and provides a richer conversation experience through real-time voice and image generation.

[1110] "Personal data collection means" refers to means for collecting data related to specific individuals, such as publicly available information and content posted on social networking services.

[1111] "Generative AI" is an artificial intelligence technology for generating natural conversations based on collected data.

[1112] The "voice recognition / synthesis means" refers to a means including a voice recognition technology that converts voice data into text data and a voice synthesis technology that converts text data into voice data.

[1113] "Image recognition / synthesis means" refers to means including image recognition technology that analyzes image data and extracts specific information, and image synthesis technology that generates a new image.

[1114] The "emotion recognition means" is a means for analyzing and recognizing the user's emotions from the user's voice, facial expressions, etc.

[1115] The "means for generating conversation according to emotions" refers to a means for generating appropriate conversation content based on the recognized emotions of the user.

[1116] This invention is a system for fulfilling the desire of ordinary consumers to converse with celebrities or the deceased, and includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and an emotion-based conversation generation means.

[1117] Server Operation

[1118] The server first uses the personal data collection means to collect personal data such as public information of celebrities and deceased people and posts on social networking services. This collection is carried out using a web crawler or an API. For example, the server uses the API to collect past posts of celebrities.

[1119] The server then pre-processes the collected personal data, filtering out unnecessary information and cleaning up the text data, removing advertisements and spam from collected posts, and normalizing the text using natural language processing techniques.

[1120] The preprocessed data is then input into a generative AI system to generate conversations that mimic the characteristics of celebrities and deceased individuals. A common natural language generation technique is used as the generative AI model for this generation. For example, the server inputs a prompt sentence such as "How was your day today?" and generates a response from the celebrity that reads, "Today was a very good day. My new project was a success."

[1121] The generated speech is then converted to audio using speech recognition / synthesis means, such as using text-to-speech software. For example, the server may send the generated text to a text-to-speech API to generate an audio file.

[1122] Additionally, the server may use image recognition / synthesis means to generate an image of the celebrity or deceased person, using image generation technology, for example, the server may use image generation technology to generate a facial image of the celebrity and create an image file for display to the user.

[1123] Using the emotion recognition means, the server recognizes the user's emotions. It sends the user's voice and facial expression data to the emotion recognition API and judges whether the user is happy or sad.

[1124] Finally, the server uses generative artificial intelligence to generate appropriate conversations based on the user's recognized emotions. For example, if the user is sad, the server generates comforting conversations such as, "It's okay, you're not alone. I've been through similar experiences."

[1125] Terminal operation

[1126] The device receives voice and text input from the user and sends it to the server. For example, if a user says to a smartphone, "How was your day today?", the voice data is sent to the server.

[1127] When the device receives the generated conversation, image, and audio data from the server, it displays and plays it for the user. For example, the device displays the generated image of a celebrity on the screen and plays the audio from a speaker.

[1128] User Actions

[1129] Users can access the system through a terminal and enjoy the experience of conversing with celebrities or deceased people. For example, if a user starts the app on their smartphone and speaks to the terminal, asking "How was your day today?", the user can experience a dialogue with a celebrity or deceased person through the conversation, images, and audio returned from the terminal.

[1130] Examples

[1131] The user speaks to the device, saying, "How was your day today?" The device sends the speech to the server. The server uses generative artificial intelligence to generate a conversation in which the celebrity or deceased person says something like, "It was a great day today. My new project was a success." The generated conversation is converted to voice using speech recognition / synthesis means and sent to the device. The device plays the voice to the user.

[1132] Examples of prompt statements

[1133] "Generate how a celebrity would respond if a user asked, 'How was your day?'"

[1134] In this way, users can experience interactions with celebrities and deceased people.

[1135] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1136] Step 1:

[1137] The server uses the personal data collection means to collect personal data such as public information of celebrities and deceased people and content posted on social networking services. Specifically, the server acquires data using a web crawler or an API. For example, the server uses an API to collect past posts of celebrities. The input is the URL or API endpoint to be collected, and the output is the collected text data.

[1138] Step 2:

[1139] The server preprocesses the collected personal data. It filters unnecessary information and cleans the text data. Specifically, it removes advertisements and spam from the collected posts and normalizes the text using natural language processing techniques. The input is the collected text data, and the output is the preprocessed clean text data.

[1140] Step 3:

[1141] The server inputs the preprocessed data into a generative AI system to generate a conversation that imitates the characteristics of celebrities or deceased people. A common natural language generation technique is used as the generative AI model. For example, the server inputs a prompt sentence such as "How was your day today?" and generates a response from the celebrity such as "Today was a very good day. My new project was successful." The input is the prompt sentence and the preprocessed text data, and the output is the generated conversation text.

[1142] Step 4:

[1143] The server converts the generated dialogue into audio using speech recognition / synthesis means. For this conversion, the server uses voice synthesis software. For example, the server sends the generated text to a speech synthesis API to generate an audio file. The input is the generated dialogue text and the output is the audio file.

[1144] Step 5:

[1145] The server uses image recognition / synthesis means to generate images of celebrities and deceased individuals. The image generation uses image generation technology. For example, the server uses image generation technology to generate facial images of celebrities and create image files to display to users. The input is the generated conversation text and related image data, and the output is the generated image file.

[1146] Step 6:

[1147] The server recognizes the user's emotions using emotion recognition means. It sends the user's voice and facial expression data to the emotion recognition API and judges whether the user is happy or sad. The input is the user's voice data and facial expression images, and the output is the recognized emotion data.

[1148] Step 7:

[1149] The server uses generative artificial intelligence to generate appropriate conversations according to the user's recognized emotions. For example, if the user is sad, the server generates a comforting conversation such as, "Don't worry, you're not alone. I've been through similar experiences." The input is the recognized emotion data and a prompt sentence, and the output is a conversation text according to the emotion.

[1150] Step 8:

[1151] The terminal receives voice and text input from the user and sends it to the server. For example, when a user speaks to a smartphone, "How was your day today?", the voice data is sent to the server. The input is the user's voice and text input, and the output is the transmission of data to the server.

[1152] Step 9:

[1153] The terminal receives the generated conversation, image, and audio data from the server. For example, the terminal displays the generated image of a celebrity on the screen and plays the audio from a speaker. The input is the data from the server, and the output is the display and playback to the user.

[1154] Step 10:

[1155] Users access the system through a terminal and enjoy the experience of conversing with celebrities or deceased people. For example, if a user starts the app on their smartphone and speaks to the terminal, asking "How was your day today?", the user can experience a dialogue with a celebrity or deceased person through the conversation, images, and audio returned from the terminal. The input is the user's operation, and the output is the dialogue experience.

[1156] (Application example 1)

[1157] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1158] Although there are systems that allow ordinary consumers to fulfill their desire to converse with celebrities or deceased people, these systems have the problem that they cannot provide dialogue that responds to the user's emotions or an interactive experience in a virtual store. In addition, there is a lack of technology that can track the user's location and movement to enable dialogue with celebrities or deceased people in real time. This may limit the user experience and reduce satisfaction.

[1159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1160] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion engine, an interactive display means for interacting with celebrities and deceased people in a virtual store, and a location recognition means for tracking the user's location and movement. This enables conversations according to the user's emotions and interactive experiences in the virtual store, improving the user experience and satisfaction.

[1161] "Means of collecting personal data" refers to means of collecting personal data such as publicly available information about celebrities or the deceased, or the contents of social media posts.

[1162] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of celebrities or deceased people based on collected personal data.

[1163] "Speech recognition / synthesis means" refers to a means for recognizing a user's utterances and converting the generated conversation into voice.

[1164] "Image recognition / synthesis means" means means for generating images of celebrities or deceased persons and tracking the user's location and movements.

[1165] An "emotion engine" is a means by which a generative AI recognizes the user's emotions and generates conversations based on those emotions.

[1166] An "interactive display means for interacting with celebrities or deceased people in a virtual store" is a means for displaying celebrities or deceased people in a virtual store and interactively interacting with the user.

[1167] "Location awareness means for tracking a user's location and movements" means means for recognizing a user's location and movements in real time and providing an interactive experience.

[1168] The system for implementing this invention is configured using the following hardware and software. The hardware used is a smartphone and a head-mounted display (HMD). The software used is Unity (game engine), TensorFlow (generative AI model), Google Cloud Speech-to-Text (voice recognition), Google Cloud Text-to-Speech (voice synthesis), and OpenCV (image recognition).

[1169] The server includes personal data collection means, conversation generation means using generative AI, voice recognition / synthesis means, image recognition / synthesis means, an emotion engine, interactive display means for interacting with celebrities and deceased people in the virtual store, and location recognition means for tracking the user's location and movements.

[1170] First, the personal data collection means collects personal data such as public information of celebrities and deceased people and their social media posts. The collected data is analyzed using natural language processing (NLP) technology to extract important keywords and phrases.

[1171] Next, a generative AI-based conversation generator generates conversations that mimic the characteristics of celebrities or deceased people based on the collected personal data. The emotion engine adjusts the conversations to the user's emotions. For example, if the user is feeling happy, the generative AI generates conversations in which the celebrity or deceased person shares their joy.

[1172] The speech recognition / synthesis method uses Google Cloud Speech-to-Text to convert user speech into text in real time, and then converts the generated conversation into audio using Google Cloud Text-to-Speech.

[1173] The image recognition / synthesis method uses OpenCV to track the user's position and movements, generate 3D models of celebrities and deceased people within Unity, and display them interactively in response to the user's movements.

[1174] As a concrete example, when a user approaches the music section of a virtual store, a deceased famous musician appears and begins a conversation about the musician's life and music. When the user asks, "What is your favorite song?", the generative AI responds, "My favorite song is 'XX'," based on information obtained from the musician's interviews and social media posts.

[1175] An example of a prompt sentence is as follows:

[1176] User: What is your favorite song?

[1177] Generative AI model: My favorite song is "XX". This song has a special meaning in my life.

[1178] In this way, users can enjoy interacting with celebrities and deceased people within the virtual store.

[1179] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1180] Step 1:

[1181] The server uses the personal data collection means to collect personal data such as publicly available information and social media posts about celebrities and deceased people. As input, a web scraping tool (e.g., BeautifulSoup) is used to analyze the collected data using natural language processing (NLP) techniques to extract important keywords and phrases. As output, the analyzed personal data is obtained.

[1182] Step 2:

[1183] The server uses a generative AI-based conversation generation means to generate a conversation that mimics the characteristics of celebrities and deceased people based on the personal data collected in step 1. As input, the analyzed personal data is used to generate a conversation with a generative AI model (e.g., TensorFlow). As output, the generated conversation text is obtained.

[1184] Step 3:

[1185] The server uses an emotion engine to recognize the user's emotion and adjust the conversation generated in step 2 according to the user's emotion. As input, the server uses the user's emotion data and the generated conversation text to adjust the conversation in the emotion engine. As output, the adjusted conversation text is obtained.

[1186] Step 4:

[1187] The device uses speech recognition / synthesis to convert the user's speech into text in real time. As input, it uses the user's voice data and converts it into text using Google Cloud Speech-to-Text. As output, it obtains the text of the user's speech.

[1188] Step 5:

[1189] The server synthesizes the adjusted conversation text from step 3. As input, it uses the adjusted conversation text to generate speech data using Google Cloud Text-to-Speech. As output, it obtains the generated speech data.

[1190] Step 6:

[1191] The device tracks the user's location and movement using image recognition / synthesis. As input, it uses camera data and recognizes the user's location and movement using OpenCV. As output, it obtains the user's location data.

[1192] Step 7:

[1193] The server uses Unity to generate 3D models of celebrities and deceased people based on the user's location data obtained in step 6, and displays them interactively in the virtual store. The server uses the user's location data and the 3D model data of celebrities and deceased people as inputs, and generates interactive displays in Unity. The output is an interactive display in the virtual store.

[1194] Step 8:

[1195] Users enjoy interacting with celebrities and deceased people in a virtual store. As input, the system uses the user's speech and movements to generate dialogue and interact with the person through interactive displays. As output, the user's satisfaction is improved.

[1196] Example 2

[1197] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1198] In a system that allows ordinary consumers to fulfill their wish to talk to celebrities or deceased people, conventional technology had the problem that it was not possible to generate conversations or synthesize voice and images in real time, and it was not possible to respond according to the user's emotions. In addition, there was also the problem that the generated conversations, voices, and images were not natural, limiting the user experience.

[1199] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1200] In this invention, the server includes a personal data collection means, a conversation generation means using a generative AI model, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and an emotion-based voice and image generation means, which enables real-time conversation generation according to the user's emotions and the provision of natural voice and images.

[1201] "Personal data collection means" refers to a means for converting personal data provided by a user, such as letters, diaries, testimonies, etc., into digital format and storing it on a server.

[1202] A "generative AI model" is an artificial intelligence model that generates conversations that mimic the characteristics of a specific person based on collected personal data.

[1203] A "conversation generation means" is a means for generating a conversation based on collected personal data using a generative AI model.

[1204] The "voice recognition / synthesis means" refers to a means for converting the generated conversation data into voice and generating a voice that resembles the voice of a specific person.

[1205] An "image recognition / synthesis means" is a means for generating a 3D model from photographic data of a specific person and generating an image of that person.

[1206] The "emotion recognition means" is a means for analyzing the user's voice and facial expressions and recognizing the user's emotions.

[1207] The "means for generating voice and images according to emotions" refers to means for generating appropriate voice and images according to the user's emotions recognized by the emotion recognition means.

[1208] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1209] This invention is a system for enabling ordinary consumers to fulfill their desire to converse with celebrities or deceased people. This system includes a personal data collection means, a conversation generation means using a generative AI model, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion recognition means, and an emotion-based voice and image generation means.

[1210] Personal data collection methods

[1211] Users collect personal data such as past letters from grandparents or celebrities, diaries, and testimonials from family members. These data are converted into digital format using hardware such as scanners and cameras and stored on the terminal. The terminal then uploads the stored digital data to the server.

[1212] Conversation generation using generative AI models

[1213] The server receives the uploaded personal data and stores it in a database. It then uses a generative AI model (e.g., a general-purpose natural language processing model) to generate a conversation based on the personal data. This generative AI model analyzes the collected data and generates a natural conversation that mimics the characteristics of a specific person.

[1214] Speech recognition / synthesis means

[1215] The server sends the generated conversation data to a voice recognition / synthesis means (e.g., general-purpose voice synthesis software). The voice recognition / synthesis means converts the conversation data into a voice that resembles the voice of a specific person. The generated voice data is temporarily stored on the server.

[1216] Image recognition / synthesis methods

[1217] The server sends photo data of a specific person to an image recognition / synthesis means (e.g., general-purpose image generation technology). The image recognition / synthesis means generates a 3D model from the photo data and creates an image of the person. The generated image data is temporarily stored on the server.

[1218] emotion recognition means

[1219] While the user is using the system, the server transmits the user's voice and facial expression in real time to an emotion recognition means (e.g., general-purpose emotion analysis software). The emotion recognition means analyzes the user's emotions and sends feedback to the server.

[1220] Means for generating sounds and images according to emotions

[1221] The server issues instructions to the voice recognition / synthesis means and the image recognition / synthesis means based on feedback from the emotion recognition means to generate appropriate voice and image. For example, if the user feels surprised, the voice recognition / synthesis means generates a surprised voice, and the image recognition / synthesis means generates a surprised image.

[1222] Examples

[1223] Example 1: Conversation with grandparents

[1224] If a user wants to experience a conversation with their grandparents, they can input the following prompt into the generative AI model:

[1225] "Grandpa, please tell me some of your memories from the past."

[1226] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the grandparents' characteristics. It then uses a speech recognition / synthesis means to generate a voice that resembles the grandparents' voice, and an image recognition / synthesis means to generate an image of the grandparents.

[1227] Example 2: Conversation with a celebrity

[1228] If a user wants to experience a conversation with a celebrity, they can input the following prompt into the generative AI model:

[1229] "Celebrities, tell us about your accomplishments."

[1230] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the celebrity's characteristics, then uses a speech recognition / synthesis method to generate a voice similar to the celebrity's voice, and an image of the celebrity is generated using an image recognition / synthesis method.

[1231] The system allows users to experience realistic conversations with deceased people or celebrities.

[1232] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1233] System program processing flow

[1234] Step 1: Collecting personal data

[1235] Users collect personal data such as past letters from grandparents or celebrities, diaries, testimonials from family members, etc. This data is converted into digital form using hardware such as scanners and cameras and stored on the device.

[1236] Input: Physical data such as letters, diaries, testimonies, etc.

[1237] Output: Personal data in digital format

[1238] Step 2: Upload your data

[1239] The terminal uploads the stored digital data to a server.

[1240] Input: Personal data in digital form

[1241] Output: Digital data stored on a server

[1242] Step 3: Conversation generation

[1243] The server receives the uploaded personal data, stores it in a database, and then uses a generative AI model to generate a conversation based on the personal data.

[1244] Input: Personal data in digital form

[1245] Output: Generated conversation data

[1246] Step 4: Text-to-speech

[1247] The server transmits the generated conversation data to a voice recognition / synthesis means, which converts the conversation data into a voice that resembles the voice of a specific person.

[1248] Input: Generated conversation data

[1249] Output: Generated audio data

[1250] Step 5: Image Generation

[1251] The server transmits photographic data of a particular person to the image recognition / synthesis means, which generates a 3D model from the photographic data and creates an image of the person.

[1252] Input: Photo data of a specific person

[1253] Output: Generated image data

[1254] Step 6: Emotion Recognition

[1255] While the user is using the system, the server transmits the user's voice and facial expression in real time to the emotion recognition means, which analyzes the user's emotions and sends feedback to the server.

[1256] Input: User's voice and facial expression data

[1257] Output: Parsed emotion data

[1258] Step 7: Emotion-based voice and image generation

[1259] The server issues instructions to the voice recognition / synthesis means and the image recognition / synthesis means based on feedback from the emotion recognition means to generate appropriate voice and image. For example, if the user feels surprised, the voice recognition / synthesis means generates a surprised voice, and the image recognition / synthesis means generates a surprised image.

[1260] Input: Parsed emotion data

[1261] Output: Voice data and image data according to emotions

[1262] Examples

[1263] Example 1: Conversation with grandparents

[1264] If a user wants to experience a conversation with their grandparents, they can input the following prompt into the generative AI model:

[1265] "Grandpa, please tell me some of your memories from the past."

[1266] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the grandparents' characteristics. It then uses a speech recognition / synthesis means to generate a voice that resembles the grandparents' voice, and an image recognition / synthesis means to generate an image of the grandparents.

[1267] Example 2: Conversation with a celebrity

[1268] If a user wants to experience a conversation with a celebrity, they can input the following prompt into the generative AI model:

[1269] "Celebrities, tell us about your accomplishments."

[1270] The server receives the prompt and uses a generative AI model to generate a conversation that mimics the celebrity's characteristics, then uses a speech recognition / synthesis method to generate a voice similar to the celebrity's voice, and uses an image recognition / synthesis method to generate an image of the celebrity.

[1271] The system allows users to experience realistic conversations with deceased people or celebrities.

[1272] (Application example 2)

[1273] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1274] Conventional technology has made it difficult for ordinary consumers to experience conversations with deceased people or celebrities. In particular, providing a realistic conversation experience that corresponds to the user's emotions is technically complex and difficult to achieve. Furthermore, there has been no system that can easily provide such an experience using a mobile terminal such as a smartphone. As a result, users have been unable to enjoy a realistic conversation experience that corresponds to their emotions.

[1275] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1276] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, an emotion engine, a means for generating voice and images according to the user's emotion, and an application installed on a smartphone, which allows the user to experience a real conversation with a deceased person or a celebrity.

[1277] "Personal data collection means" are means for collecting personal information of deceased persons or celebrities, such as past letters or diaries, or testimonies from family members.

[1278] A "conversation generation method using generative AI" is a method that uses artificial intelligence technology to generate conversations that imitate the characteristics of a deceased person or celebrity based on collected personal data.

[1279] The "voice recognition / synthesis means" refers to a means for recognizing the user's voice and converting the generated conversation into voice.

[1280] "Image recognition / synthesis means" means means for generating images using 3D models generated from photographs of deceased persons or celebrities.

[1281] An "emotion engine" is a means for recognizing a user's emotions and generating sounds and images according to those emotions.

[1282] "Means for generating voice and images according to the user's emotions" refers to means for controlling voice recognition / synthesis means and image recognition / synthesis means based on the emotions felt by the user, and generating appropriate voice and images.

[1283] The "application installed on a smartphone" is an application that runs on a smartphone and allows the user to enjoy the experience of having a conversation with a deceased person or a celebrity.

[1284] The system for implementing this invention uses the following hardware and software. The hardware requires a smartphone, a microphone, and a speaker. The software uses Python, OpenAI API, Google Text-to-Speech (gTTS), OpenCV, and SpeechRecognition.

[1285] The server uses a personal data collection means to collect personal information of the deceased or famous person, such as past letters and diaries, testimonials from family members, etc. This collected data is analyzed by a conversation generation means using generative AI, and conversations that imitate the characteristics of the deceased or famous person are generated.

[1286] The terminal (smartphone) uses a voice recognition / synthesis means to recognize the user's voice and convert the generated conversation into audio.Furthermore, it uses an image recognition / synthesis means to generate an image using a 3D model generated from a photograph of a deceased person or a celebrity.

[1287] The emotion engine recognizes the user's emotion and generates voice and image according to the emotion. For example, if the user feels surprised, the voice recognition / synthesis means generates a voice that sounds surprised, and the image recognition / synthesis means generates an image that looks surprised. On the other hand, if the user feels relieved, the voice recognition / synthesis means generates a voice that sounds relieved, and the image recognition / synthesis means generates an image that looks relieved.

[1288] As a concrete example, if a user says, "What happened today?", the emotion engine recognizes the user's emotion as "surprise." In this case, the generative AI generates a conversation using the following prompt sentence:

[1289] Example prompt:

[1290] “Generate conversations when users have an emotion of surprise: What happened today?”

[1291] By inputting this prompt sentence into the generative AI model, a conversation corresponding to the emotion of surprise is generated. The generated conversation is converted into voice by the speech recognition / synthesis means, and an image with a surprised facial expression is generated by the image recognition / synthesis means.

[1292] As described above, the present invention is a system that enables a user to experience a realistic conversation with a deceased person or a celebrity.

[1293] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1294] Step 1:

[1295] The server uses the personal data collection means to collect personal information such as past letters and diaries of deceased people and celebrities, and testimonials from family members. The input is personal data such as letters, diaries, and testimonials, and the output is a database of the collected personal data. Specifically, the server saves the data in the database and organizes and classifies the data as necessary.

[1296] Step 2:

[1297] The server analyzes the collected personal data using a generative AI-based conversation generation means to generate conversations that mimic the characteristics of deceased people or celebrities. The input is the collected personal data, and the output is the generated conversation text. Specifically, the server inputs a prompt sentence into the generative AI model to generate the conversation text.

[1298] Step 3:

[1299] The terminal (smartphone) recognizes the user's voice using a speech recognition / synthesis means. The input is the user's voice, and the output is the recognized text. Specifically, the voice is captured using a microphone and converted to text using the SpeechRecognition library.

[1300] Step 4:

[1301] The terminal uses an emotion engine to recognize the user's emotion. The input is the recognized text, and the output is the recognized emotion. Specifically, the terminal performs text analysis and estimates the emotion.

[1302] Step 5:

[1303] The server uses generative AI to generate a conversation according to the user's emotions. The input is the recognized emotion and the user's text, and the output is a conversation text according to the emotion. Specifically, it generates a prompt sentence and inputs it into the generative AI model to generate the conversation text. An example of a prompt sentence is "Generate a conversation when the user has an emotion of surprise: What happened today?"

[1304] Step 6:

[1305] The terminal uses a speech recognition / synthesis means to convert the generated conversation text into speech. The input is the generated conversation text, and the output is an audio file. Specifically, the terminal uses Google Text-to-Speech (gTTS) to convert the text into speech and generate an audio file.

[1306] Step 7:

[1307] The device uses image recognition / synthesis to generate images using 3D models generated from photos of deceased people or celebrities. The input is the recognized emotion and the 3D model, and the output is an image corresponding to the emotion. Specifically, the device uses OpenCV to generate an image with a facial expression corresponding to the emotion.

[1308] Step 8:

[1309] The terminal provides the generated audio and images to the user. The input is an audio file and an image, and the output is the audio and images provided to the user. In concrete terms, the terminal operates by playing the audio using a speaker and displaying the images using a display.

[1310] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1311] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1312] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1313] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1314] [Fourth embodiment]

[1315] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1316] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1317] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1318] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1319] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1320] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1321] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1322] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1323] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1324] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1325] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1326] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1327] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1328] "Example 1"

[1329] As one embodiment of the present invention, a system is provided that allows general consumers to experience conversations with celebrities and deceased persons. This system is configured by combining a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, and an image recognition / synthesis means. The personal data collection means collects personal data such as public information of celebrities and deceased persons and SNS posts. The generative AI generates a conversation that imitates the characteristics of the celebrity or deceased person based on the collected personal data. The voice recognition / synthesis means converts the generated conversation into voice. The image recognition / synthesis means generates an image of the celebrity or deceased person. By combining these means, a system is realized that allows general consumers to experience conversations with celebrities and deceased persons.

[1330] "Example 2"

[1331] As a specific example, consider a system that allows general consumers to experience a conversation with their deceased grandparents. First, a personal data collection means collects personal data such as the grandparents' past letters and diaries, and testimonies from family members. A generative AI generates conversation that mimics the characteristics of the grandparents based on the collected personal data. A voice recognition / synthesis means synthesizes the generated conversation into a voice that is close to the grandparents' voice. An image recognition / synthesis means generates images of the grandparents using 3D models generated from photos of the grandparents. By combining these means, a system is realized that allows general consumers to experience a conversation with their deceased grandparents.

[1332] The process flow of each embodiment will be described below.

[1333] "Example 1"

[1334] Step 1: Personal data such as publicly available information and social media posts about celebrities and deceased people is collected using personal data collection methods.

[1335] Step 2: Generative AI generates conversations that mimic the characteristics of celebrities or deceased people based on the collected personal data.

[1336] Step 3: The speech recognition / synthesis means converts the generated speech into audio.

[1337] Step 4: Image recognition / synthesis procedures generate an image of the celebrity or deceased person.

[1338] "Example 2"

[1339] Step 1: Personal data collection instruments are used to collect personal data such as grandparents' past letters and diaries, and testimonies from family members.

[1340] Step 2: The generative AI generates conversations that mimic the characteristics of the grandparents based on the collected personal data.

[1341] Step 3: The speech recognition / synthesis means synthesizes the generated conversation into a voice that resembles the grandparents' voice.

[1342] Step 4: The image recognition / synthesis means generates images of the grandparents using the 3D models generated from the photos of the grandparents.

[1343] Example 1

[1344] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1345] There is a demand for a more realistic and natural conversation experience in a system that allows ordinary consumers to fulfill their wish to talk to celebrities or deceased people. However, conventional systems have limitations in the quality of collected data, the naturalness of generated conversations, and the realism of voice and images, resulting in an insufficient user experience.

[1346] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1347] In this invention, the server includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition means, a voice synthesis means, an image recognition means, and an image generation means, which enable the server to generate a conversation that imitates the characteristics of a specific person based on the collected personal data, convert a user's voice input into text, vocalize the generated conversation, and recognize and generate an image of the specific person.

[1348] "Personal data collection means" refers to means for collecting data such as publicly available information about a specific person and the contents of social media posts.

[1349] "Generative AI" is AI that generates conversations that mimic the characteristics of a specific person based on collected data.

[1350] A "speech recognition means" is a means for converting a user's voice input into text.

[1351] The "voice synthesis means" is a means for converting the generated conversation into voice.

[1352] The "image recognition means" is a means for recognizing an image of a specific person.

[1353] An "image generating means" is a means for generating an image of a particular person.

[1354] This invention is a system for fulfilling the desire of ordinary consumers to converse with celebrities or deceased people, and includes a personal data collection means, a conversation generation means using generative artificial intelligence, a voice recognition means, a voice synthesis means, an image recognition means, and an image generation means.

[1355] Personal data collection methods

[1356] The server collects personal data such as public information and social media posts of celebrities and deceased people. This data is collected using web scraping tools (e.g., Beautiful Soup and Scrapy) and social media APIs (e.g., Twitter API and Facebook Graph API). The collected data is stored in a database.

[1357] Conversation Generation Method

[1358] The server generates a conversation using generative artificial intelligence (e.g., OpenAI's GPT-4) based on the collected personal data. The generative artificial intelligence analyzes the input personal data and generates natural conversational sentences. An example of a prompt sentence is, "Based on the public information and SNS posts of famous singer A, please generate a conversation that imitates his characteristics."

[1359] Speech recognition and synthesis means

[1360] The server converts the generated conversational text into voice using speech synthesis software (e.g., Google Text-to-Speech or Amazon Polly). It also converts the user's voice input into text using speech recognition software (e.g., Google Speech-to-Text or IBM Watson Speech to Text). This allows the user's question to be obtained as text and input into the generative artificial intelligence.

[1361] Image recognition and image generation

[1362] The server uses image generation software (e.g. DALL-E or DeepArt) to generate images of celebrities or deceased people, allowing the user to visually recognize the person they are speaking with. It inputs prompts to the image generation software based on the collected personal data, and sends the generated image to the user's device.

[1363] User interaction

[1364] Users access the system through a terminal (e.g., a smartphone or PC) and experience conversations with celebrities or deceased people. When the user asks a question by voice, the voice recognition means converts the question into text, and the generative artificial intelligence generates an answer. The generated answer is converted into voice by the voice synthesis means and returned to the user. At the same time, an image of the celebrity or deceased person, generated by the image recognition means and the image generation means, is displayed.

[1365] Examples

[1366] For example, if a user wishes to "talk to famous singer A," the system operates as follows.

[1367] 1. The server collects public information and social media posts about famous singer A.

[1368] 2. The server uses generative artificial intelligence based on the collected data to generate conversations that mimic the characteristics of famous singer A.

[1369] 3. The server converts the generated conversational text into voice using speech synthesis software.

[1370] 4. The server generates an image of famous singer A.

[1371] 5. The user experiences a conversation with famous singer A through the terminal using audio and images.

[1372] In this way, a system is realized that allows ordinary consumers to experience conversations with celebrities and deceased people.

[1373] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1374] Step 1:

[1375] The server uses personal data collection means to collect public information and social media posts about celebrities and deceased people. Specifically, it uses web scraping tools (e.g. Beautiful Soup and Scrapy) and social media APIs (e.g. Twitter API and Facebook Graph API). The input is the name and related information of a specific person, and the output is the collected personal data. The collected data is stored in a database.

[1376] Step 2:

[1377] The server retrieves the collected personal data from the database and generates a conversation using generative artificial intelligence (e.g., OpenAI's GPT-4). The input is the collected personal data, and the output is the generated conversation. Specifically, a prompt sentence is input into the generative artificial intelligence, and a conversation sentence that imitates the characteristics of a specific person is generated. An example of a prompt sentence is, "Based on the public information of famous singer A and his social media posts, please generate a conversation that imitates his characteristics."

[1378] Step 3:

[1379] The server converts the generated conversation text into voice using speech synthesis software (e.g., Google Text-to-Speech or Amazon Polly). The input is the generated conversation text, and the output is an audio file. Specifically, the generated conversation text is input into the speech synthesis software to generate an audio file.

[1380] Step 4:

[1381] The server converts the user's voice input into text using voice recognition software (e.g., Google Speech-to-Text or IBM Watson Speech to Text). The input is the user's voice, and the output is text. Specifically, the user's voice input is input into the voice recognition software and converted into text.

[1382] Step 5:

[1383] The server generates an image of a specific person using image generation software (e.g., DALL-E or DeepArt) based on the collected personal data. The input is the collected personal data, and the output is the generated image. Specifically, a prompt is input to the image generation software, and an image of a specific person is generated.

[1384] Step 6:

[1385] Users access the system through a terminal (e.g., a smartphone or PC) and experience a conversation with a celebrity or a deceased person. The input is the user's voice question, and the output is the generated audio and image of the conversation. Specifically, when the user asks a question by voice, the voice recognition means converts the question into text, and the generative artificial intelligence generates an answer. The generated answer is converted into voice by the voice synthesis means and returned to the user. At the same time, an image of the celebrity or deceased person, generated by the image recognition means and the image generation means, is displayed.

[1386] (Application example 1)

[1387] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1388] When ordinary consumers wish to converse with celebrities or deceased people, it is difficult to provide a realistic experience that would allow them to realize their wish. In particular, advanced technology is required to provide a real-time conversation experience that combines voice and images. There is also a demand to provide this experience in a form that is easy for users to use.

[1389] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1390] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, an application means installed on a smartphone, a means for recognizing a user's voice input and generating a voice response in real time, and a means for generating an image of a celebrity or a deceased person and displaying it during a conversation. This allows general consumers to easily enjoy a real conversation experience with a celebrity or a deceased person.

[1391] "Personal data collection means" refers to means for collecting personal data, such as publicly available information about celebrities or the deceased, or the content of social media posts.

[1392] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of celebrities or deceased people based on collected personal data.

[1393] "Speech recognition / synthesis means" means for recognizing a user's voice input and converting the generated speech into voice.

[1394] "Image recognition / synthesis means" refers to means for generating images of celebrities or deceased people and displaying them during a conversation.

[1395] The "application means installed on a smartphone" refers to a means that is installed on a smartphone and provides an application that enables a user to have a conversation experience with a celebrity or a deceased person.

[1396] The "means for recognizing a user's voice input and generating a voice response in real time" refers to a means for recognizing a user's voice input in real time and generating a voice response in response thereto.

[1397] "Means for generating an image of a celebrity or a deceased person and displaying it during a conversation" refers to means for generating an image of a celebrity or a deceased person and displaying it to a user during a conversation.

[1398] The system for carrying out the present invention is configured as follows. First, the server uses a personal data collection means to collect personal data such as public information and social media posts of celebrities and deceased people. This collected data is input to a conversation generation means using generative AI, and conversations that imitate the characteristics of celebrities and deceased people are generated.

[1399] Next, an application installed on the user's device (smartphone) uses speech recognition / synthesis means to recognize the user's voice input and generates a real-time voice response to convert the generated conversation into voice. It also uses image recognition / synthesis means to generate images of celebrities and deceased people and display them to the user during the conversation.

[1400] Specifically, the server uses the following software and hardware: It uses the OpenAI API for generative AI, the SpeechRecognition library for voice recognition, and the gTTS library for voice synthesis. It uses the OpenCV library for image processing. These software run on the server and work in conjunction with applications installed on the user's smartphone.

[1401] For example, if a user wants to talk to a "historical figure," the server collects personal data about that person and uses generative AI to generate a conversation that mimics the characteristics of that person. When the user asks a question to the smartphone, the voice recognition / synthesis means recognizes the question and generates a voice response in real time. At the same time, the image recognition / synthesis means generates an image of the person and displays it on the smartphone screen.

[1402] An example of a prompt is as follows:

[1403] "Collect personal data about a historical figure and generate answers to the following user questions that mimic the characteristics of that person.

[1404] User: What is your dream?

[1405] AI:

[1406] In this way, users can easily enjoy the experience of having a real conversation with a celebrity or a deceased person.

[1407] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1408] Step 1:

[1409] The server uses the personal data collection means to collect personal data such as public information and social media posts of celebrities and deceased individuals. The input is the name of the celebrity or deceased individual, and the output is the collected personal data. This data collection is done through public information on the Internet and social media APIs.

[1410] Step 2:

[1411] The server inputs the collected personal data into a generative AI to generate a conversation that imitates the characteristics of celebrities and deceased people. The input is the collected personal data, and the output is the generated conversation text. The generative AI uses OpenAI's API to generate a conversation based on the prompt sentence.

[1412] Step 3:

[1413] The user's device (smartphone) recognizes the user's voice input using a speech recognition / synthesis means. The input is the user's voice and the output is the user's question in text form. The SpeechRecognition library is used for speech recognition.

[1414] Step 4:

[1415] The server inputs the user's question text into the generative AI, which generates a voice response in real time. The input is the user's question text, and the output is the generated response text. The generative AI again uses OpenAI's API to generate a response based on the prompt text.

[1416] Step 5:

[1417] The user's terminal inputs the generated response text to a speech synthesis means and converts it into speech. The input is the generated response text, and the output is an audio file. The gTTS library is used for speech synthesis.

[1418] Step 6:

[1419] The server generates images of celebrities and deceased people using image recognition / synthesis methods. The input is the name of the celebrity or deceased person, and the output is the generated image file. The OpenCV library is used for image generation.

[1420] Step 7:

[1421] The user's terminal displays the generated audio and image files. The input is the audio and image files, and the output is visual and auditory feedback to the user. The audio files are played through the speaker, and the image files are shown on the display.

[1422] Example 2

[1423] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1424] Conventional technology has made it difficult for ordinary consumers to experience conversations with deceased people or specific individuals. In particular, generating natural conversations that reflect the characteristics of the deceased and realistic voices and images is technically complex and difficult to achieve. For this reason, there is a demand for a system that can reproduce conversations with the deceased.

[1425] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1426] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a means for providing a conversation experience by combining the generated voice and image, thereby enabling general consumers to experience a natural conversation with a deceased person or a specific person.

[1427] "Personal data collection means" refers to devices or methods for collecting and digitizing personal information of a specific individual, such as past letters, diaries, and testimonies.

[1428] A "conversation generation method using generative AI" is an artificial intelligence technology for generating conversations that mimic the characteristics of a specific individual based on collected personal data.

[1429] "Speech recognition / synthesis means" refers to technology for converting the generated conversation into voice data and generating a voice that resembles the voice of a specific person.

[1430] "Image recognition / synthesis means" is a technology for generating 3D models and realistic images based on photographs or videos of specific people.

[1431] "Means for combining generated audio and images to provide a conversational experience" refers to an apparatus or method for synchronizing generated audio data and image data to provide a user with an interactive conversational experience.

[1432] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1433] This invention is a system for enabling general consumers to experience conversations with deceased people or specific individuals. The system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a means for providing a conversation experience by combining the generated voice and image.

[1434] Collection of Personal Data

[1435] Users collect personal data such as past letters from grandparents or specific individuals, diaries, and testimonials from family members. These data are digitized using a scanner, voice recording device, and text input software. For example, a user can use a scanner to scan letters from grandparents and save them as text data.

[1436] Conversation generation using generative AI

[1437] The server receives the collected personal data and uses a generative AI model (e.g., a general natural language processing model) to generate a conversation that mimics the characteristics of grandparents or a specific person. Specifically, the server analyzes the data and inputs a prompt sentence into the generative AI model. For example, the server inputs a prompt sentence such as, "Based on the contents of the letter from my grandparents, please generate a conversation that reflects the characteristics of my grandparents."

[1438] Speech Recognition / Synthesis

[1439] The server uses a voice recognition / synthesis means (for example, a general voice synthesis API) to synthesize the generated conversation into a voice that resembles the voice of a grandparent or a specific person. Specifically, the server sends the generated text data to a voice synthesis API and obtains a voice file. For example, the generated conversation, "It's a very nice day today," is synthesized into the voice of a grandparent.

[1440] Image Recognition / Synthesis

[1441] The server uses image recognition / synthesis means (e.g., a general image generation algorithm) to generate a 3D model using photos of the grandparents or a specific person. Specifically, the server analyzes the photo data and inputs it into a 3D model generation algorithm. For example, a 3D model is generated based on a photo of the grandparents' faces, and an avatar that moves in real time is created.

[1442] Providing a conversational experience

[1443] The device combines the generated voice and image to provide the user with a conversation experience with a grandparent or a specific person. Specifically, the device plays the audio file and the 3D model in sync, allowing the user to have an interactive conversation. For example, the user can talk to an avatar of a grandparent while looking at the screen of the device.

[1444] Examples and prompts

[1445] Examples

[1446] A user scans a letter from their grandparents and uploads it to the system. The server analyzes the contents of the letter and uses a generative AI model to generate a conversation that mimics the characteristics of the grandparents. The generated conversation is then converted into voice using a common voice synthesis API, and a 3D model of the grandparent is generated using a common image generation algorithm. Finally, the user can talk to the grandparents through the device.

[1447] Examples of prompt statements

[1448] "Based on the content of the letter below, please generate a conversation that reflects the characteristics of the grandparents. The content of the letter is as follows: 'Dear Grandson, It's a very nice day today and the flowers in the garden are in full bloom. If you were here, we could go for a walk together.'"

[1449] By inputting this prompt into the generative AI model, natural conversation that reflects the characteristics of the grandparents is generated.

[1450] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1451] Step 1:

[1452] The user collects personal data such as past letters, diaries, and testimonials from family members of grandparents or other specific individuals. These data are digitized using a scanner, a voice recording device, and text input software. Specifically, the user uses a scanner to scan letters from grandparents and saves them as text data. The input is physical personal data such as letters and diaries, and the output is digitized text data.

[1453] Step 2:

[1454] The server receives digitized personal data uploaded by the user. It then uses a generative AI model (e.g., a general natural language processing model) to generate a conversation that mimics the characteristics of a specific person. Specifically, the server analyzes the data and inputs a prompt sentence into the generative AI model. For example, the server inputs the prompt sentence, "Based on the contents of my grandparents' letters, please generate a conversation that reflects my grandparents' characteristics." The input is the digitized personal data, and the output is the generated conversation text.

[1455] Step 3:

[1456] The server synthesizes the generated conversation text into a voice that resembles the voice of a specific person using a voice recognition / synthesis means (for example, a general voice synthesis API). Specifically, the server sends the generated text data to a voice synthesis API and obtains an audio file. For example, the generated conversation "It's a very nice day today" is synthesized in the voice of a grandparent. The input is the generated conversation text, and the output is an audio file.

[1457] Step 4:

[1458] The server uses image recognition / synthesis means (e.g., a general image generation algorithm) to generate a 3D model using a photo of a specific person. Specifically, the server analyzes the photo data and inputs it into a 3D model generation algorithm. For example, a 3D model is generated based on a photo of a grandparent's face, and an avatar that moves in real time is created. The input is photo data of a specific person, and the output is a 3D model.

[1459] Step 5:

[1460] The terminal combines the generated audio file with the 3D model to provide the user with a conversation experience with a specific person. Specifically, the terminal plays the audio file and the 3D model in sync, allowing the user to have an interactive conversation. For example, a user can interact with an avatar of their grandparents while looking at the screen of the terminal. The input is the audio file and the 3D model, and the output is an interactive conversation experience.

[1461] (Application example 2)

[1462] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1463] There are existing systems that allow ordinary consumers to fulfill their desire to converse with deceased relatives or celebrities, but these systems have the problem of lacking realism and limiting the user experience. In addition, there is a lack of means to realize dialogue in a virtual space, making it difficult to provide a more realistic experience. This has led to a demand for a system that allows users to experience a more realistic conversation with deceased relatives or celebrities.

[1464] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1465] In this invention, the server includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a display means for realizing a conversation in a virtual space, which allows general consumers to experience a more realistic conversation with a deceased person or a celebrity.

[1466] "Personal data collection methods" are methods for collecting data about individuals, such as past letters, diaries, and testimonies from family members of deceased or famous people.

[1467] A "conversation generation method using generative AI" is a method for generating conversations that imitate the characteristics of deceased people or celebrities based on collected personal data.

[1468] The "voice recognition / synthesis means" refers to a means for converting the generated conversation into voice and synthesizing a voice that is close to the voice of the deceased or famous person.

[1469] "Image recognition / synthesis means" refers to a means for generating 3D models from photographs or videos of deceased persons or celebrities and synthesizing realistic images.

[1470] "Display means for realizing dialogue in a virtual space" refers to a means for displaying generated conversations and images in a virtual space, providing a user with the experience of interacting with a deceased person or a celebrity.

[1471] The following system configuration will be described as an embodiment of the present invention.

[1472] System Configuration

[1473] This system includes a personal data collection means, a conversation generation means using generative AI, a voice recognition / synthesis means, an image recognition / synthesis means, and a display means for realizing dialogue in a virtual space.

[1474] Personal data collection methods

[1475] The server collects data about deceased or famous people, such as past letters, diaries, and testimonials from family members. This data may be provided by users or collected from public sources on the Internet.

[1476] Conversation generation using generative AI

[1477] Based on the collected personal data, the server uses generative AI to generate conversations that mimic the characteristics of deceased people and celebrities. Specifically, it uses OpenAI's API to generate conversations based on prompt sentences.

[1478] Speech recognition / synthesis means

[1479] The server uses a speech synthesis engine (e.g., pyttsx3) to convert the generated conversation into audio and synthesize a voice that is close to the voice of the deceased person or celebrity, allowing users to experience the conversation with a realistic voice.

[1480] Image recognition / synthesis methods

[1481] The server uses image processing libraries (e.g., OpenCV) to generate 3D models from photos and videos of deceased people and celebrities and synthesize realistic images, allowing users to experience realistic interactions.

[1482] Display means for realizing dialogue in virtual space

[1483] The device displays the generated conversations and images in a virtual space, allowing users to have the experience of interacting with deceased people or celebrities. Specifically, the device uses a smartphone and a head-mounted display to realize the interaction in the virtual space.

[1484] Examples

[1485] For example, consider the case where a user uploads a letter from their grandparents to an app and experiences the generated conversation. The user can talk to their grandparents in a virtual space using a smartphone or a head-mounted display. In this case, the following prompt sentence is used:

[1486] Examples of prompt statements

[1487] "Generate conversations by mimicking your grandparents' traits: Grandparents' letters, diaries, and family testimonials."

[1488] In this way, consumers can have a more realistic conversation experience with deceased people or celebrities.

[1489] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1490] Step 1:

[1491] Users use the device to upload personal data (letters, ...

Claims

1. A system that allows ordinary consumers to fulfill their desire to converse with celebrities or deceased people, A means of collecting personal data including public information of celebrities and deceased persons and past posts on social networking services; A means for generating a prompt sentence instructing the generation of an answer by a famous person or a deceased person to a question from a user; A means for generating an answer of the celebrity or the deceased to a question from a user by using the generated prompt sentence, the personal data of the celebrity or the deceased collected by the collection means, and a generation AI model; a voice synthesis means for vocalizing the generated answers using voice synthesis software in a conversational manner that mimics the characteristics of the famous person or deceased person; A system including:

2. The collection means performs preprocessing by removing advertisements and spam from the collected posts of celebrities or deceased people, and converts the preprocessed posts into text data using natural language processing technology, and collects the text data as personal data. The system of claim 1 .

3. The speech synthesis means uses the generative AI model to generate speech that mimics the characteristics of a celebrity or deceased person based on the collected personal data. The system of claim 1 .

Images