System

Abstract

To provide a novel experience that enables a user to meet not only a person from the past but also a person from the future.SOLUTION: A system comprises: means for acquiring an image of a person a user wishes to meet again, uploaded by the user; means for generating computer graphics reproducing the face of the person by using a prompt sentence that instructs reproduction of the face of the person using computer graphics on the basis of the image of the person, and a generative AI model; means for accepting an instruction from the user to meet the person in the future or past; means for modifying the generated computer graphics by using a prompt sentence that instructs application of age transformation to the generated computer graphics on the basis of the accepted instruction, and the generative AI model; and means for displaying the modified computer graphics.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] One of the challenges is the difficulty of visually recreating and interacting with people you want to reunite with. In particular, it is difficult with current technology to provide the new experience of meeting people not only from the past but also from the future. [Means for solving the problem]

[0005] To solve this problem, we provide a means to understand the personality of the person you want to reunite with and to recreate a realistic interaction based on that personality. Furthermore, we provide a means to visually recreate the person, allowing you to meet not only past but also future contacts. This allows users to experience realistic interactions with the person they want to reunite with, making it possible to meet not only past but also future contacts. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, the coded communication I / F (Interface) refers to a communication processor and

[0013] The communication I / F is an interface that includes a network interface (NIC) and an antenna. The communication I / F controls communication between multiple computers. Examples of communication standards that can be applied to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0014] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0015] [First embodiment]

[0016] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0026] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0027] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0028] "Example 1"

[0029] In one embodiment of the present invention, information provided by the user (e.g., photos, videos, audio, text information, etc.) is collected as a means of understanding the personality of the person the user wishes to reunite with, and an image of the person is generated based on this information using artificial intelligence (AI). The AI ​​learns the person's appearance, speaking style, personality, preferences, etc., and generates an image of the person based on this information.

[0030] "Example 2"

[0031] In addition, as a means of recreating realistic interactions, the AI ​​will converse with the user by imitating the other person's speaking style and tone based on the generated persona. For example, if the user greets the user with "Good morning," the AI ​​will respond with "Good morning" in the other person's tone.

[0032] "Example 3"

[0033] Furthermore, as a means of visually recreating the other person, CG technology is used to mimic the other person's appearance based on the generated image. For example, based on a photo of the other person, AI can recreate their face using CG and display it to the user.

[0034] "Example 4"

[0035] To allow users to meet their partner not only in the past but also in the future, the AI ​​will add or subtract their age and change their appearance. For example, if a user requests to meet their partner in 10 years, the AI ​​will apply the changes to their appearance in 10 years' time and display the results to the user.

[0036] The processing flow of each embodiment will be described below.

[0037] "Example 1"

[0038] Step 1: Collect information from the user about the person they wish to reunite with (e.g., photos, videos, audio, text information, etc.).

[0039] Step 2: Using the collected information, artificial intelligence (AI) is used to generate a profile of the person. The AI ​​learns the person's appearance, speech pattern, personality, preferences, etc.

[0040] "Example 2"

[0041] Step 1: Based on the generated image, the AI ​​imitates the other person's speaking style and tone.

[0042] Step 2: Dialogue with the user. For example, if the user says "Good morning," the AI ​​will respond with "Good morning" in the user's voice.

[0043] "Example 3"

[0044] Step 1: Using the generated image, CG technology is used to mimic the appearance of the other person.

[0045] Step 2: For example, based on a photo of the other person, the AI ​​will recreate their face in CG and display it to the user.

[0046] "Example 4"

[0047] Step 1: The AI ​​adds or subtracts the opponent's age and changes their appearance.

[0048] Step 2: For example, if a user says, "I want to meet my partner 10 years from now," the AI ​​will apply changes to the partner's appearance in 10 years' time and display the results to the user.

[0049] Example 1

[0050] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0051] There is a demand for a system that can provide a detailed understanding of the personality of a person with whom a user wishes to reunite and allow the user to recreate a realistic interaction with that person. However, conventional technologies have had difficulty in comprehensively analyzing the other person's appearance, speech style, personality, preferences, etc. to generate an accurate personality profile. Furthermore, there has been a lack of means to effectively provide the generated personality profile to the user. As a result, users have been unable to obtain detailed information about the person they wish to reunite with, and have been unable to have a satisfying reunion experience.

[0052] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0053] In this invention, the server includes means for collecting information provided by users, means for analyzing the collected information, means for generating a person's profile based on the analyzed information, and means for providing the generated person's profile to the user, thereby enabling the user to accurately understand the detailed profile of the person they wish to reunite with and to recreate a realistic interaction.

[0054] The "means of collecting information provided by users" refers to a function that allows users to upload photos, videos, audio, text information, etc. related to the person they wish to reunite with to the system, which is then received and stored by the server.

[0055] "Means for analyzing collected information" refers to a function that analyzes the photos, videos, audio, and text information collected by the server and extracts characteristics such as the other person's appearance, speaking style, personality, and preferences.

[0056] "Image recognition technology" is a technology for analyzing the external characteristics of others from photographs and videos, and includes algorithms that detect face shape, hairstyle, clothing, etc.

[0057] "Speech recognition technology" is a technology that converts voice data into text and further analyzes the tone and speaking characteristics of the voice.

[0058] "Natural language processing technology" is a technology for analyzing text data and inferring the personality and preferences of others, and involves the use of generative AI models.

[0059] The "means for generating a person's profile" is a function that creates a detailed profile of the other person by integrating the other person's appearance, speaking style, personality, preferences, etc. based on analyzed information.

[0060] The "means for providing the generated portrait to the user" is a function for notifying the user of the generated portrait on the user's terminal and displaying the profile.

[0061] This invention is a system for enabling a user to understand in detail the personality of a person with whom they wish to reunite and to recreate realistic interactions. This system includes a series of processes that collect and analyze information provided by the user, generate a personality profile of the other person, and provide it to the user.

[0062] First, users upload photos, videos, audio, and text information about the person they want to reunite with to the system. Users can easily provide this information using a smartphone application. For example, users can take photos of the person they want to reunite with and record and upload audio messages.

[0063] The server then collects the information provided by the user and stores it in a database. Specifically, the server uses a cloud storage service such as an AWS® S3 bucket to store photos and audio files.

[0064] The collected information is analyzed by the server. Using image recognition technology, the server analyzes photos and videos using libraries such as OpenCV and TENSORFLOW (registered trademark) to extract the other person's external characteristics, such as facial shape, hairstyle, and clothing.

[0065] Using voice recognition technology, the server converts the voice data into text using voice recognition services such as Google® Speech-to-Text API or IBM Watson®, and further analyzes the voice tone and speaking characteristics.

[0066] Using natural language processing technology, the server uses generative AI models such as GPT-3 (registered trademark) and BERT to analyze text extracted from text information and audio data and infer the other person's personality and preferences.

[0067] Based on these analysis results, the server generates a profile of the other person, including their appearance, speaking style, personality, and preferences. For example, it creates a profile that says, "The other person has short hair, speaks softly, and enjoys the outdoors."

[0068] Finally, the server provides the generated profile to the user. Specifically, the server sends a notification to the user's device and displays the generated profile. The user can then open the application and check the detailed profile of the other person.

[0069] An example of a prompt is as follows:

[0070] "You provide a photo and an audio message of the person you want to reunite with. Based on this information, we analyze their appearance, speaking style, personality, and preferences to generate a profile of them."

[0071] In this way, the user can accurately understand the detailed personality of the person they wish to meet again and recreate a realistic interaction.

[0072] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0073] Step 1: User Provides Information

[0074] Users upload photos, videos, audio, text information, etc. of the person they want to reunite with to the system. Specifically, users use a smartphone application to take photos of the person they want to reunite with and record and upload audio messages. The input is the photos, videos, audio, and text information provided by the user, and the output is the transmission of this data to the server.

[0075] Step 2: The server collects information

[0076] The server receives the information provided by the user and stores it in a database. Specifically, the server uses cloud storage services such as AWS S3 buckets to store photos and audio files. The input is the photos, videos, audio, and text information sent by the user, and the output is that this data is stored in cloud storage.

[0077] Step 3: The server analyzes the image

[0078] The server uses image recognition libraries such as OpenCV and TensorFlow to analyze the stored photos and videos. Specifically, the server runs a face detection algorithm to extract the facial features of the other person (eye position, nose shape, mouth shape, etc.). The input is the photos and videos stored in cloud storage, and the output is the extracted facial feature data.

[0079] Step 4: The server analyzes the audio

[0080] The server uses speech recognition services such as Google Speech-to-Text API and IBM Watson to analyze the stored voice data. Specifically, the server converts the voice data into text and further extracts voice tone and speaking characteristics. The input is the voice data stored in cloud storage, and the output is text data and voice feature data.

[0081] Step 5: The server parses the text

[0082] The server uses natural language processing models such as GPT-3 and BERT to analyze the text information provided by the user. Specifically, the server analyzes the text data and infers the other person's personality and preferences. The input is the text information stored in cloud storage, and the output is the analyzed personality and preference data.

[0083] Step 6: The server integrates the information and generates a profile

[0084] The server integrates features extracted from images, voice, and text information to generate a profile of the other person. Specifically, the server compiles the other person's appearance, speaking style, personality, and preferences into a single profile. The input is the extracted facial feature data, voice feature data, personality and preference data, and the output is the generated profile of the person.

[0085] Step 7: The server provides the generated profile to the user

[0086] The server provides the generated persona to the user. Specifically, the server sends a notification to the user's device and displays the generated profile. The input is the generated persona profile, and the output is the profile displayed on the user's device.

[0087] (Application example 1)

[0088] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] In modern society, it is not easy for people to find someone they want to reunite with. This is especially true in large spaces such as large brick-and-mortar stores and shopping malls. It is also difficult to accurately reproduce the appearance, manner of speech, personality, and other characteristics of the person you want to reunite with and have a realistic interaction. This often leads to missed reunion opportunities, which leads to a decline in user satisfaction.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0091] In this invention, the server includes means for understanding the personality of a person the user wishes to reunite with, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for searching for the person in a physical store, means for notifying the user when the person is found, and means for allowing the user to meet not only past but also future partners. This allows the user to easily find the person they wish to reunite with even in a large physical store, and allows for realistic interactions that accurately reproduce the person's characteristics.

[0092] "A means of understanding the personality of the person you want to reunite with" is a technology in which AI learns the other person's appearance, speaking style, personality, preferences, etc. based on information provided by the user (photos, videos, audio, text information, etc.), and generates an image of that person.

[0093] "Means for recreating realistic interactions" refers to technology that uses voice recognition technology and natural language processing technology to imitate the speaking style and tone of the other person based on a generated image of the other person, and engage in realistic conversations with the user.

[0094] "Means of visual reproduction" refers to technology that uses image recognition technology and computer graphics technology to imitate the appearance of the other person and visually reproduce the other person for the user.

[0095] The "means of finding a partner within a physical store" refers to a technology that analyzes camera footage within a physical store in real time and uses facial recognition technology to find the person you want to meet again with.

[0096] The "means for notifying when the person is found" refers to a technology that sends a notification to a device such as smart glasses or a smartphone when a person you want to reunite with is found, informing the user.

[0097] "A means to meet not only past but also future partners" is a technology that allows you to save information about not only past partners you want to meet again with, but also people you want to meet again with in the future, and then reunite with those people in the future.

[0098] The system for implementing this invention has the function of understanding the profile of the person the user wants to meet, searching for that person in a physical store, and notifying the user when the person is found. A specific embodiment of this system will be described below.

[0099] First, the user provides information about the person they want to reunite with. This information includes photos, videos, audio, and text of the person. This information is sent to a server and used as training data for an AI model. The server uses facial recognition libraries such as DeepFace and face_recognition to learn the person's appearance, speaking style, personality, preferences, etc., and generates a profile of that person.

[0100] Next, to find the person in the physical store, the server analyzes the camera footage in the physical store in real time. It uses image processing libraries such as OpenCV to detect faces from the camera footage and identifies the person using facial recognition technology. The server then matches the detected face with an AI model to find the person you want to reunite with.

[0101] When a match is found, the server sends a notification to the device, such as smart glasses or a smartphone. The notification informs the user that the match has been found and offers an opportunity to reunite. The notification is displayed on the smart glasses display or the smartphone's notification function.

[0102] For example, a user enters a photo and characteristics of a person they would like to meet in a shopping mall into the smart glasses. The server trains an AI model based on the provided information and analyzes camera footage in the mall to find the person. When the person is found, the smart glasses display a notification saying "Partner Found!"

[0103] Example prompt sentence:

[0104] Develop an application that allows users to provide photos, videos, audio, and text information about the person they wish to reunite with, and uses AI to generate a profile of that person. The application will be installed on smart glasses and will help users find their match in a physical store. Specifically, the AI ​​will generate a profile of the person based on the information provided by the user, and then locate that person in the store in real time.

[0105] In this way, users can easily find the person they want to meet again even in a large physical store, and can have a realistic interaction that accurately reproduces the other person's characteristics.

[0106] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0107] Step 1:

[0108] The user provides information about the person they want to meet. The user inputs photos, videos, audio, and text information about the other person into their device. This information is then sent to the server. The input data includes characteristics such as the other person's appearance, speaking style, personality, and preferences.

[0109] Step 2:

[0110] The server trains an AI model based on the information provided. The server uses facial recognition libraries such as DeepFace and face_recognition to learn the other person's appearance. It also uses voice recognition technology to learn the other person's speaking style and tone. This generates a profile of the other person. The input data is the information provided by the user, and the output data is the trained AI model.

[0111] Step 3:

[0112] The server analyzes camera footage from inside the physical store in real time. The server uses image processing libraries such as OpenCV to detect faces from the camera footage. The detected faces are compared with an AI model. The input data is the camera footage from inside the physical store, and the output data is the location information of the detected faces.

[0113] Step 4:

[0114] The server uses facial recognition technology to identify the person you want to reunite with. The server then matches the detected face with an AI model to find the person. The input data is the location information of the detected face, and the output data is the result of identifying the person.

[0115] Step 5:

[0116] The server sends a notification when the other party is found. The server then sends a notification to a device such as smart glasses or a smartphone to notify the user. The notification includes a message indicating that the other party has been found. The input data is the result of identifying the other party, and the output data is the notification message.

[0117] Step 6:

[0118] The user receives a notification and has the opportunity to reunite. The user checks the notification displayed on the smart glasses or smartphone and takes action to reunite with the other person. The input data is the notification message, and the output data is the user's action.

[0119] Example 2

[0120] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] With current communication technology, it is difficult to realistically recreate the image of someone you want to meet again. Furthermore, there is a lack of means to interact realistically with not only past contacts but also future contacts. Furthermore, technology to generate natural dialogue based on user input is not yet fully developed. To solve these issues, a system is needed that can mimic the speaking style and tone of the other person and visually recreate them.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0123] In this invention, the server includes means for understanding the personality of a person the user wishes to reunite with, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting not only past but also future people, means for receiving input from the user, means for converting the received input into a prompt sentence, means for generating a response from the prompt sentence using a generative AI model, and means for displaying the generated response to the user, thereby enabling the user to have a realistic conversation with past or future people.

[0124] "Means to understand the personality of the person you would like to meet again" refers to technology for collecting and analyzing data such as the characteristics, personality, and speaking style of people you have met in the past or people you would like to meet in the future.

[0125] "Means of recreating real interactions" refers to technology that mimics the speaking style and tone of the other person based on collected personalities, allowing for natural dialogue with the user.

[0126] "Means of visual reproduction" refers to techniques that use image recognition technology and computer graphics technology to realistically reproduce the appearance and facial expressions of the other person.

[0127] "A means to meet people not only from the past but also in the future" is a technology that can predict and reproduce the characteristics of not only people you have met in the past, but also people you would like to meet in the future.

[0128] The "means for receiving input from the user" is an interface for sending text or voice input by the user to the server.

[0129] "Means for converting received input into prompt text" refers to technology for converting text or speech received from a user into a format that is easy for a generative AI model to understand.

[0130] "Means for generating a response from a prompt sentence using a generative AI model" refers to technology for using a generative AI model to generate an appropriate response based on a prompt sentence.

[0131] "Means for displaying the generated response to the user" means an interface for visually or audibly presenting the response obtained from the generative AI model to the user.

[0132] This invention is a system that enables users to have realistic conversations with people from the past or future. This system operates among three parties: a server, a terminal, and a user.

[0133] Hardware and software used

[0134] Hardware:

[0135] A server equipped with a high-performance GPU (e.g., NVIDIA Tesla V100)

[0136] software:

[0137] Generative AI models (e.g., GPT-4 (registered trademark))

[0138] System Operation Overview

[0139] server:

[0140] The server receives input from the user, converts it into a prompt, uses a generative AI model to generate a response from the prompt, and sends the response to the device for display to the user.

[0141] Device:

[0142] The terminal transmits the text entered by the user to the server and displays the response received from the server to the user.

[0143] user:

[0144] The user enters text into the terminal and checks the response from the server.

[0145] Specific examples

[0146] Take the example of a user typing "good morning" into a terminal.

[0147] 1. Receiving user input:

[0148] The user types "Good morning" into the chat application on their device.

[0149] The terminal transmits the entered text to the server in real time.

[0150] 2. Generate prompt sentence:

[0151] Based on the received text "Good morning," the server generates a prompt sentence: "When the user says 'Good morning,' please reply in the other person's tone of voice."

[0152] 3. Generative AI model generates responses:

[0153] The server inputs the generated prompt sentence into the generative AI model, which analyzes the prompt sentence and generates the appropriate response, "Good morning."

[0154] 4. Sending a response:

[0155] The server sends the response "Good morning" obtained from the generative AI model to the device.

[0156] 5. User Visibility:

[0157] The device receives the reply "Good morning" from the server and displays it on the chat application screen, allowing the user to see the reply imitating the other person's speaking style and tone.

[0158] In this way, users can have realistic conversations with people from the past or future. The system uses generative AI models to create natural conversations based on user input.

[0159] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0160] Step 1: Receiving User Input

[0161] User: Type text into the device's chat application (e.g. "Good morning").

[0162] Terminal: Receives text entered by the user and sends it to the server.

[0163] Input: The text entered by the user (e.g., "Good morning").

[0164] Output: The text data sent to the server.

[0165] Step 2: Generate a prompt statement

[0166] Server: Generates a prompt based on the received user text.

[0167] Specific operation: The server analyzes the received text "Good morning" and generates a prompt sentence: "When the user says 'Good morning,' please reply in the other person's tone of voice."

[0168] Input: The text received from the user (e.g. "Good morning").

[0169] Output: The generated prompt (e.g., "When the user says 'Good morning,' please respond in the other person's voice tone.").

[0170] Step 3: Generative AI model generates a response

[0171] Server: Inputs the generated prompt sentence into the generative AI model and generates a response.

[0172] Specific operation: The server inputs a prompt sentence into a generative AI model (e.g., GPT-4), which analyzes it and generates the appropriate response, "Good morning."

[0173] Input: The generated prompt (e.g., "When the user says 'Good morning,' please respond in their voice.").

[0174] Output: The response obtained from the generative AI model (e.g., "Good morning").

[0175] Step 4: Send your response

[0176] Server: Sends the answers obtained from the generative AI model to the device.

[0177] Specific operation: The server sends the generated response "Good morning" to the terminal.

[0178] Input: A response obtained from a generative AI model (e.g., "Good morning").

[0179] Output: The response data sent to the device.

[0180] Step 5: Display to the user

[0181] Terminal: Displays the response received from the server to the user.

[0182] Specific operation: The device displays the reply "Good morning" received from the server on the chat application screen.

[0183] Input: Response data received from the server (e.g. "Good morning").

[0184] Output: The response that is displayed to the user (e.g. "Good morning").

[0185] In this way, users can experience realistic interactions.

[0186] (Application example 2)

[0187] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0188] In conventional customer service, it was difficult to provide personalized service based on customer preferences and past purchase history, making it difficult to improve customer satisfaction. In addition, the way a salesperson speaks and their tone of voice have a significant impact on customers, so there was a need for a way to improve the customer experience by imitating the speaking style and tone of a specific salesperson.

[0189] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for understanding the personality of a person the customer wants to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting not only past but also future people, means for providing customer service by imitating the customer's speaking style and tone, means for supporting dialogue with the customer using smart glasses, and means for imitating the customer's speaking style and tone using a generative AI model. This enables personalized customer service based on the customer's preferences and past purchase history, and makes it possible to improve the customer experience by imitating the speaking style and tone of a specific store clerk.

[0190] "A means of understanding the personality of the person you wish to reunite with" is a technology that analyzes the characteristics, personality, and past interactions of a specific person with whom the user wishes to reunite, and saves this data.

[0191] "Means of recreating realistic interactions" refers to technology that uses stored character data to mimic actual conversations and behaviors, enabling natural communication with users.

[0192] "Means of visual reproduction" refers to techniques that use image recognition technology and CG technology to realistically reproduce the appearance and facial expressions of a specific person.

[0193] "A means to meet people not only from the past but also from the future" is a technology that generates images of people who may exist in the future as well as people who existed in the past, allowing users to interact with them.

[0194] "Means of serving customers by imitating the way a customer speaks and the tone of their speech" refers to a technique of analyzing the way a customer speaks and the tone of their speech and imitating it to serve customers.

[0195] "Means for supporting conversations with customers using smart glasses" refers to technology that allows store clerks wearing smart glasses to support conversations with customers.

[0196] "Means of using a generative AI model to mimic a customer's speaking style and tone" refers to a technology that uses a generative AI model to mimic a customer's speaking style and tone, enabling natural dialogue.

[0197] A system for implementing this invention includes means for understanding the personality of a person you wish to reunite with, means for recreating real interactions, means for visually recreating interactions, means for meeting not only past but also future counterparts, means for serving customers by imitating the speaking style and tone of voice of the customer, means for supporting interactions with customers using smart glasses, and means for imitating the speaking style and tone of voice of the customer using a generative AI model.

[0198] Hardware and software used

[0199] Hardware: Smart glasses (with built-in camera and microphone)

[0200] Software: OpenAI (registered trademark) API, OpenCV, pyttsx3

[0201] System Operation

[0202] The server first analyzes the information provided by the user to understand the person they want to reunite with. This information includes specific person characteristics and past interactions. The analyzed data is then stored using a generative AI model.

[0203] To replicate realistic interactions, a generative AI model is then used to generate natural-looking dialogue based on the stored persona data, allowing users to interact with a specific person in a realistic way.

[0204] The visual reproduction is achieved by using image recognition and CG technology to realistically reproduce the appearance and facial expressions of a specific person, allowing users to visually feel as if they are interacting with that person.

[0205] Furthermore, as a means to meet people not only from the past but also from the future, the technology generates images of people who may exist in the future and allows users to interact with them. This technology is realized using a generative AI model.

[0206] As a means of serving customers by imitating their speaking style and tone, store clerks wearing smart glasses can analyze the speaking style and tone of the customer and imitate them when serving them, allowing customers to receive more personalized service.

[0207] The smart glasses are used to support customer interactions by capturing the customer's voice input in real time using the glasses' camera and microphone, and then using a generative AI model to generate responses that mimic the customer's speaking style and tone. The generated responses are then played back aloud using pyttsx3.

[0208] Specific examples

[0209] For example, if a customer says "Good morning," the smart glasses' microphone captures the audio and sends it to the generative AI model, which then uses the following prompt sentence to generate a response that mimics the customer's speaking style and tone:

[0210] Example prompt sentence:

[0211] Respond by imitating the customer's speaking style and tone: Good morning

[0212] The generated response is played back aloud using pyttsx3 in the form of "Good morning, what product are you looking for today?" This allows customers to experience a natural dialogue that mimics the speaking style and tone of a specific store clerk.

[0213] In this way, a system for implementing the invention can provide personalized customer service based on customer preferences and past purchase history, and can enhance the customer experience by mimicking the speaking style and tone of a particular store associate.

[0214] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0215] Step 1:

[0216] The server receives the information provided by the user and understands the profile of the person they wish to reunite with. This information includes the specific person's characteristics and past interactions. The server analyzes this information and inputs it into a generative AI model. The generative AI model generates and stores data on the profile of the person based on the input information.

[0217] Input: User-provided personal characteristics and past interactions

[0218] Data processing: Analysis and data generation using generative AI models

[0219] Output: Saved person image data

[0220] Step 2:

[0221] The server recreates realistic interactions based on the stored persona data, using a generative AI model to generate natural dialogue with the user, allowing the user to have realistic interactions with a specific person.

[0222] Input: Saved portrait data

[0223] Data Computation: Dialogue Generation with Generative AI Models

[0224] Output: The generated dialogue

[0225] Step 3:

[0226] The server uses image recognition and CG technology to visually recreate the appearance and facial expressions of a specific person, giving the user the feeling that they are visually interacting with that person.

[0227] Input: Saved portrait data

[0228] Data processing: Reproducing appearance and facial expressions using image recognition and CG technology

[0229] Output: A visually recreated human figure

[0230] Step 4:

[0231] The server generates a future image of a person that may exist, allowing the user to interact with it. The server uses a generative AI model to generate and store the future image.

[0232] Input: Information about future persona

[0233] Data Computation: Generating Future Humans with Generative AI Models

[0234] Output: Saved future persona data

[0235] Step 5:

[0236] The device allows store clerks wearing smart glasses to analyze the customer's speaking style and tone and imitate it to serve the customer. The smart glasses' microphone captures the customer's voice and sends it to a generative AI model. The generative AI model then generates a response that imitates the customer's speaking style and tone and plays it back aloud using pyttsx3.

[0237] Input: Customer voice input

[0238] Data processing: Analyzing and imitating speaking style and tone with generative AI models

[0239] Output: Audio playback of the imitated response

[0240] Step 6:

[0241] The device uses the smart glasses' camera and microphone to capture the customer's voice input in real time, then uses a generative AI model to generate responses that mimic their speaking style and tone, which are then played back aloud using pyttsx3.

[0242] Input: Customer voice input captured in real time

[0243] Data Computation: Generative AI models mimic speaking style and tone and generate responses

[0244] Output: Audio playback of the imitated response

[0245] In this way, the system can personalize customer interactions based on customer preferences and past purchases, and improve the customer experience by mimicking the speaking style and tone of a particular store associate.

[0246] Example 3

[0247] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0248] In modern society, it is physically impossible to reunite with someone from the past or future. However, advances in technology have made it possible to recreate the appearance and speaking style of a person. The present invention aims to provide a system that allows users to experience visual and auditory reunions with people from the past or future.

[0249] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for acquiring an image of a person the user wishes to reunite with, uploaded by the user; means for generating computer graphics that reproduce the person's face using a generative AI model and a prompt statement instructing the user to recreate the person's face in computer graphics based on the image of the person; means for accepting an instruction from the user that the user wishes to meet a future or past person; means for modifying the generated computer graphics using a prompt statement instructing the user to change the age of the generated computer graphics based on the accepted instruction and the generative AI model; and means for displaying the modified computer graphics. This allows the user to visually reunite with past or future people. The server also includes a means for acquiring voice data of the other person whom the user wishes to reunite with, a means for analyzing the other person's speaking style or tone from the other person's voice data, a prompt sentence instructing the user to converse with the user based on input from the user and the analysis results of the other person's speaking style or tone, a means for recreating the conversation between the user and the other person using the generative AI model, and a means for outputting the other person's remarks in the recreated conversation.

[0250] "Means for understanding the personality of the person you wish to reunite with" refers to technology that allows users to recognize the appearance and characteristics of the person they wish to reunite with and capture them as data.

[0251] "Means for reproducing realistic interactions" refers to technology that imitates the other person's speaking style, tone of voice, facial expressions, etc., to realize a dialogue with the user.

[0252] "Means of visually reproducing the other person" refers to technology that allows the appearance of the other person to be reproduced as an image or video.

[0253] "A means to meet not only past but also future counterparts" is a technology that adds or subtracts the other person's age and generates their appearance in the past or future.

[0254] "Means for users to upload photos of other users" refers to technology that allows users to use their terminals to send photos of other users to the system.

[0255] "Means for the server to receive photos and input them into the generative AI model" refers to technology that enables the server to receive photos sent by users and input them into the generative AI model in an appropriate format.

[0256] "Means for a generative AI model to recreate the other person's face using CG" refers to a technology that enables a generative AI model to recreate the other person's face using highly accurate CG based on input photographic data.

[0257] The "means for a user to indicate that he / she wishes to meet a future or past partner" is a technique for a user to input an instruction to the system that he / she wishes to meet a future or past partner.

[0258] "Means for the server to receive instructions and instruct the generative AI model to make age changes" refers to technology that enables the server to receive instructions from the user and instruct the generative AI model to perform age change calculations.

[0259] "Means by which a generative AI model generates the appearance of a partner with age changes" refers to technology that enables a generative AI model to perform age addition or subtraction calculations to generate the future or past appearance of a partner.

[0260] "Means for displaying the appearance generated by the server to the user" refers to a technique for transmitting the appearance of the other party generated by the server to the user's terminal and displaying it.

[0261] This invention is a system that allows users to experience a visual and auditory reunion with a past or future partner. In this system, the user uploads a photo of the partner, and the server receives the photo and inputs it into a generative AI model, which recreates the partner's face in CG. Furthermore, when the user indicates that they would like to meet a future or past partner, the server instructs the generative AI model to add an age change, and the generative AI model generates the partner's appearance with the age change added. Finally, the server displays the generated appearance to the user.

[0262] Specifically, the following hardware and software are used.

[0263] 1. User's device

[0264] Users access the system interface using a device such as a smartphone or PC, which has an application installed for uploading photos.

[0265] 2. Server

[0266] The server receives photos uploaded by users and inputs them into a generative AI model for image processing (e.g., StyleGAN or DeepFaceLab). The server converts the photo data into an appropriate format and passes it to the generative AI model.

[0267] 3. Generative AI Models

[0268] The generative AI model generates computer graphics that recreate the other person's face based on a prompt that instructs the model to recreate the other person's face in high-precision CG using input photo data. Specifically, it uses AI technologies such as StyleGAN and DeepFaceLab to extract facial features and generate the CG.

[0269] 4. Age Change Algorithm

[0270] The generative AI model incorporates algorithms that perform age addition or subtraction calculations, allowing it to generate the future or past appearance of a person. Based on the received instructions, the generative AI model modifies the generated computer graphics using the generative AI model in accordance with a prompt statement that instructs the model to make age changes to the generated computer graphics.

[0271] As a specific example, consider the case where the user indicates, "I want to meet the person in 10 years' time."

[0272] 1. Users upload photos of their friends

[0273] A user uses a smartphone to open the system's application, taps the "Upload Photo" button in the application, and selects and uploads a friend's photo from the gallery.

[0274] 2. The server receives the photo and inputs it into the generative AI model

[0275] The server receives the photo data and inputs it into StyleGAN, a generative AI model for image processing. The server converts the photo data into an appropriate format and passes it to the generative AI model.

[0276] 3. The generative AI model recreates the other person's face in CG

[0277] StyleGAN analyzes photo data and extracts facial features of your friend. Based on these extracted features, it generates a highly accurate CG image that realistically recreates your friend's face.

[0278] 4. The user indicates they want to meet a future or past contact.

[0279] The user enters "I want to meet my friend 10 years from now" into the input field in the application and taps the send button.

[0280] 5. The server receives the instructions and instructs the generative AI model to add age changes.

[0281] The server receives the user's instructions and instructs StyleGAN to apply an age-addition algorithm, specifically, to calculate how the friend's face will change in 10 years.

[0282] 6. The generative AI model generates the other person's appearance with age changes.

[0283] StyleGAN performs age addition calculations to generate what your friend will look like in 10 years. It realistically reproduces facial wrinkles and changes in hair color to generate your friend's future face.

[0284] 7. The server displays the generated appearance to the user

[0285] The server sends the generated image of the friend's appearance in 10 years to the user's smartphone, which displays the received data, allowing the user to see what the friend will look like in 10 years.

[0286] An example of the prompt sentence is "Please generate what your friend will look like in 10 years. I have uploaded a current photo." The flow of the identification process in the third embodiment will be described with reference to FIG.

[0287] Step 1:

[0288] Users upload photos of their contacts.

[0289] Input: The user uses the terminal to select the other person's photo file and upload it to the system interface.

[0290] Specific behavior: A user opens the application on their smartphone, taps the "Upload Photo" button, and selects and uploads a photo of a friend from their gallery.

[0291] Output: Uploaded photo data is sent to the server.

[0292] Step 2:

[0293] The server receives the photos and inputs them into a generative AI model.

[0294] Input: Photo data submitted by the user.

[0295] What it does: The server receives the photo data and converts it into a suitable format for input into a generative AI model (e.g., StyleGAN) for image processing.

[0296] Output: Photo data converted into a format that can be fed into a generative AI model.

[0297] Step 3:

[0298] The generative AI model recreates the other person's face in CG.

[0299] Input: Photo data sent from the server after format conversion.

[0300] How it works: A generative AI model (StyleGAN) analyzes photo data and extracts the facial features of the other person. Based on the extracted features, it generates highly accurate CG.

[0301] Output: CG data of the other person's face.

[0302] Step 4:

[0303] A user indicates that they would like to meet a future or past contact.

[0304] Input: The user inputs instructions into the system's interface, such as "I want to meet someone in the future" or "I want to meet someone in the past."

[0305] Specific behavior: The user enters "I want to meet my friend 10 years from now" into the input field in the application and taps the send button.

[0306] Output: User instructions are sent to the server.

[0307] Step 5:

[0308] The server receives the instructions and instructs the generative AI model to add age changes.

[0309] Input: A user-submitted instruction that they would like to meet a future or past contact.

[0310] Specific operation: The server receives the user's instructions and instructs the generative AI model to calculate the age change. Specifically, it applies an age addition algorithm to add changes to the other person's appearance in 10 years.

[0311] Output: Age change calculation instructions sent to the generative AI model.

[0312] Step 6:

[0313] The generative AI model generates the appearance of the other person, with age changes added.

[0314] Input: Age change calculation instructions sent from the server.

[0315] How it works: The generative AI model adds or subtracts age to generate the person's future or past appearance, realistically recreating changes in facial wrinkles, hair color, skin texture, and more.

[0316] Output: CG data of the opponent's appearance with age changes added.

[0317] Step 7:

[0318] The server displays the generated appearance to the user.

[0319] Input: CG data of the opponent's appearance with age changes sent from the generative AI model.

[0320] Specific operation: The server sends the generated CG data of the appearance to the user's device, which displays the received data.

[0321] Output: The future or past appearance of the other person displayed on the user's device.

[0322] The server also acquires the voice data of the person the user wishes to reunite with. The server may acquire voice data from the web based on the information of the person the user wishes to reunite with, or may acquire voice data uploaded by the user. The server then uses voice recognition technology to analyze the speaking style or tone of the other party from the other party's voice data. The server then uses a generative AI model (GPT-4) and prompts to instruct the user to converse with the other person based on the user's input and the analysis of the other person's speaking style or tone. The server then outputs the other person's utterances in the reproduced conversation to the user. This allows the user to reproduce a conversation with a future or past person they would like to meet.

[0323] The server may further include means for recognizing the user's emotions in the conversation using an emotion engine described below. In this case, the reproducing means reproduces the conversation between the user and the other party using the prompt sentence instructing the user to converse with the other party and the generative AI model based on the input from the user, the user's emotions, and the analysis result of the other party's speaking style or tone.

[0324] (Application example 3)

[0325] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0326] While conventional technology can recreate the image of a person you want to meet, it is difficult to display their appearance in the future or the past, or to try on clothes for the generated appearance. Furthermore, when users try on products in a virtual store, there is a lack of a way to check their appearance in the future or the past. This can prevent users from determining whether a product will suit their future self, potentially reducing their motivation to purchase.

[0327] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0328] In this invention, the server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting the person not only in the past but also in the future, means for recreating the user's appearance and adding or subtracting age to display the future or past appearance, and means for trying on clothes with the generated appearance, thereby enabling the user to try on products in a virtual store while checking their future or past appearance.

[0329] "A means of understanding the personality of the person you wish to reunite with" is a technology that recognizes the appearance and characteristics of the person you wish to reunite with and acquires this data.

[0330] "Means for recreating realistic interactions" refers to technology that mimics actual conversations and behavior based on the acquired image of the other person.

[0331] "Means of visual reproduction" refers to technology that uses image recognition technology and CG technology to visually display the appearance of the other person.

[0332] The "means of meeting not only past but also future partners" is a technology that uses AI technology to add or subtract a partner's age and generate their appearance in the past or future.

[0333] "Means of recreating a user's appearance and adding or subtracting age to display future or past appearances" refers to technology that uses AI technology to change the age based on the user's own appearance and display future or past appearances.

[0334] The "means for trying on clothes for the generated appearance" is a technology for virtually trying on selected clothes for the generated appearance.

[0335] The following system configuration is used as an embodiment of the present invention.

[0336] System Configuration

[0337] The system consists of a server, a user terminal, and various software components. The server understands the person's profile, recreates realistic interactions, and processes various data to visually recreate them. The user terminal is a device such as a smartphone or head-mounted display, and provides an interface for users to interact with the system.

[0338] Hardware and software used

[0339] Hardware: Smartphones, head-mounted displays, servers

[0340] Software: Python, OpenCV, DeepFace, PIL, ageitgey / face_recognition library

[0341] Data processing and calculation

[0342] 1. How to understand the person you want to reunite with:

[0343] The server receives the photo of the person uploaded by the user and uses image recognition technology to extract the person's appearance and features, using OpenCV and the ageitgey / face_recognition library.

[0344] 2. Means of replicating real-life interactions:

[0345] The server uses voice recognition technology to mimic the other person's speaking style and tone based on the extracted characteristics of the other person, giving the user an experience that feels like they are actually having a conversation with the other person.

[0346] 3. Visual Representation:

[0347] The server uses image recognition and CG technology to visually recreate the other person's appearance, using the DeepFace library to recreate the other person's face with high accuracy.

[0348] 4. Ways to meet not only past but future partners:

[0349] The server uses AI technology to add or subtract the partner's age to generate past and future appearances, allowing users to see what their partner will look like in the future by entering a prompt such as "I want to meet my partner in 10 years."

[0350] 5. A way to recreate the user's appearance and add or subtract age to show future or past appearances:

[0351] The server uses AI technology to age the user based on their own photo, generating future and past appearances, allowing users to see what they will look like in the future.

[0352] 6. Means of trying on clothes for generated looks:

[0353] The server virtually tries on the selected clothing for the generated look, and uses the PIL library to overlay images of the clothing onto the user's look.

[0354] Specific examples

[0355] If the user says, "I want to see how this outfit will look on me in 10 years," enter the following prompt text:

[0356] "I want to see if this outfit will look good on me in 10 years."

[0357] The server receives this prompt and generates a visual of the user's appearance in 10 years' time based on the user's current photo, and then displays the selected clothing on the visual, allowing the user to check the suitability of the product for their future self.

[0358] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0359] Step 1:

[0360] The user uses the device to upload a photo of the person they want to meet again.

[0361] Input: Photo of the person

[0362] Output: The photo data of the other person sent to the server

[0363] Specific operation: The user uses a smartphone or head-mounted display to take or select a photo of the person they want to reunite with, and uploads it to the server through the application.

[0364] Step 2:

[0365] The server analyzes the received photos and extracts the other person's appearance and characteristics.

[0366] Input: Uploaded photo data of the other person

[0367] Output: Extracted appearance and feature data of the other person

[0368] Specific operation: The server uses OpenCV and the ageitgey / face_recognition library to extract facial features from the photo and obtain data on the other person's appearance and characteristics.

[0369] Step 3:

[0370] The server uses the extracted features to recreate the other person's appearance using CG.

[0371] Input: Extracted appearance and feature data of the other person

[0372] Output: Reconstructed CG image of the other person

[0373] Specific operation: The server uses the DeepFace library to accurately recreate the other person's face based on the extracted features and generates it as a CG image.

[0374] Step 4:

[0375] A user uses a terminal to input a prompt sentence such as "I want to meet my partner in 10 years' time."

[0376] Input: Prompt sentence (e.g., "I want to meet the person in 10 years' time")

[0377] Output: Prompt text sent to the server

[0378] Specific operation: The user inputs a prompt message through the application interface indicating that they would like to meet a prospective partner and sends it to the server.

[0379] Step 5:

[0380] The server parses the prompt and adds or subtracts the person's age to generate a future or past appearance.

[0381] Input: Prompt text, CG image of the other person

[0382] Output: CG image of the other person from the future or past

[0383] What it does: The server uses a generative AI model to add or subtract the person's age based on the prompt text, generating future and past appearances.

[0384] Step 6:

[0385] The user uses the terminal to select an image of the clothing item that they would like to try on.

[0386] Input: Clothing image

[0387] Output: Image data of the garment sent to the server

[0388] Specific operation: Through the application interface, the user selects an image of the clothing they want to try on and uploads it to the server.

[0389] Step 7:

[0390] The server then tries on the selected clothing for the generated look.

[0391] Input: CG images of future or past opponents, clothing image data

[0392] Output: CG image of the person trying on the clothing

[0393] Specific operation: The server uses the PIL library to overlay an image of the clothing on the generated appearance and generates a CG image of the clothing being tried on.

[0394] Step 8:

[0395] The server sends a CG image of the garment being tried on to the user's terminal and displays it.

[0396] Input: CG image of someone trying on clothing

[0397] Output: CG image of the garment being tried on, displayed on the user's device

[0398] Specific operation: The server sends the generated CG image of the garment being tried on to the user's device, and the user can view the image via a smartphone or head-mounted display.

[0399] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0400] "Example 1"

[0401] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes the user's emotions and adjusts interactions with a person's image of a person the user wishes to reunite with based on those emotions. Specifically, the system recognizes emotions from the user's voice and facial expressions and adjusts the reaction of the other person's image based on those emotions. For example, if the user expresses joy, the system causes the other person's image to generate a joyful reaction. Conversely, if the user expresses sadness, the system causes the other person's image to generate a comforting reaction.

[0402] "Example 2"

[0403] In another embodiment of the present invention, a system is provided in which an emotion engine adjusts the future image of the other person in response to the user's emotions. Specifically, if the user expresses emotions of expectation or hope, the system adjusts the future image of the other person to be brighter and more positive. For example, if the user indicates, "I want to meet my partner in 10 years," and expresses emotions of expectation, the system displays the other person's future image in 10 years as successful and happy.

[0404] "Example 3"

[0405] Furthermore, as another embodiment of the present invention, a system is provided in which an emotion engine adjusts the past appearance of a partner according to the user's emotion. Specifically, if the user expresses feelings of nostalgia or nostalgia, the system adjusts the past appearance of the partner to be more memorable and moving. For example, if the user indicates, "I want to meet the partner five years ago," and expresses nostalgia, the system displays the partner's appearance five years ago as a special memorable scene.

[0406] The processing flow of each embodiment will be described below.

[0407] "Example 1"

[0408] Step 1: Receive input from the user (voice, facial expressions, etc.).

[0409] Step 2: The emotion engine recognizes the user's emotion.

[0410] Step 3: Tailor your interactions with the persona you want to reunite with based on the emotions you recognize.

[0411] "Example 2"

[0412] Step 1: Receive input from the user (voice, facial expressions, etc.).

[0413] Step 2: The emotion engine recognizes the user's emotion.

[0414] Step 3: If the user expresses feelings of expectation or hope, the system adjusts the other person's view of the future to be brighter and more positive.

[0415] "Example 3"

[0416] Step 1: Receive input from the user (voice, facial expressions, etc.).

[0417] Step 2: The emotion engine recognizes the user's emotion.

[0418] Step 3: If the user expresses feelings of nostalgia or nostalgia, the system adjusts the person's past appearance to make it more memorable and moving.

[0419] Example 1

[0420] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0421] With conventional technology, it was difficult to realistically recreate the image of the person you wanted to reunite with, and it was not possible to adjust interactions based on the user's emotions. In addition, there was a lack of means to meet not only past but also future contacts. This made it difficult for users to experience more realistic interactions with people they wanted to reunite with.

[0422] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0423] In this invention, the server includes means for collecting information provided by the user, means for generating a person's portrait based on the information using artificial intelligence, and means for recognizing the user's emotions and adjusting interactions with the person's portrait in accordance with the emotions. This makes it possible to realistically recreate the person the user wants to reunite with and adjust interactions in accordance with the user's emotions.

[0424] "Means for collecting information provided by users" refers to devices or software for collecting photos, videos, audio, text information, etc. of the person with whom the user wishes to reunite.

[0425] "Means for generating a person's profile using artificial intelligence" refers to algorithms or programs that learn about the other person's appearance, speaking style, personality, preferences, etc. based on collected information, and generate a profile of that person.

[0426] "Means for recognizing the user's emotions and adjusting interactions with the other person's character in accordance with those emotions" refers to a system that analyzes the user's voice and facial expressions, recognizes the user's emotions, and adjusts the other person's character's response based on those emotions.

[0427] "Means of visual reproduction" refers to devices or software that use image recognition technology and computer graphics technology to imitate and visually reproduce the appearance of another person.

[0428] "A means to meet not only past but also future partners" is a system that generates images of not only past but also future partners, allowing users to interact with those images.

[0429] "Voice recognition technology" is a technology that analyzes the voice of the user or other person and imitates their speaking style and tone.

[0430] "Image recognition technology" is a technology that analyzes the appearance of others from photographs and videos and extracts their characteristics.

[0431] "Computer graphics technology" is a technology for visually reproducing the image of a person based on analyzed external features.

[0432] The present invention provides a system that realistically recreates the image of a person with whom a user wants to reunite and adjusts interactions according to the user's emotions. Specific embodiments of this system will be described below.

[0433] First, the user provides photos, videos, audio, and text information about the person they want to reunite with. This information is collected through devices such as smartphones and PCs. The devices then send the photos taken by the user and the audio recorded by the user to a server via an application.

[0434] The server receives the information sent from the device and stores it in a database. For example, photo data is stored in an image folder, and audio data is stored in an audio folder. The server then inputs the stored information into an artificial intelligence (AI) model. This AI model may be, for example, OpenAI's GPT-4 or DeepMind's DALL-E.

[0435] The AI ​​model generates an image of the other person based on the input information. Specifically, it extracts the other person's facial features from photos and learns their speaking style from their audio. This allows it to reproduce the other person's appearance, speaking style, personality, preferences, etc.

[0436] Furthermore, an emotion engine is built in to recognize the user's emotions. The device collects the user's voice and facial expressions through a camera and microphone and sends them to the emotion engine. The emotion engine analyzes the collected voice and facial expression data to recognize the user's emotions. For example, it can determine whether the user is happy or sad from changes in voice tone and facial expression.

[0437] The server adjusts the reaction of the other person's figure based on the emotion information received from the emotion engine. For example, if the user expresses a happy emotion, the server makes the other person's figure generate a happy reaction. Conversely, if the user expresses a sad emotion, the server makes the other person's figure generate a comforting reaction.

[0438] Finally, the device displays the reaction of the other person's image received from the server to the user. For example, the other person's image may be displayed with a smiling face on the smartphone screen. In this way, the system adjusts the interaction with the other person's image according to the user's emotions.

[0439] As a concrete example, consider the case where a user provides a photo and voice of the person they want to reunite with. The user uses their smartphone to upload the photo and voice of the person to the system. The server receives this data and inputs it into an AI model. The AI ​​model learns the person's appearance and speaking style and generates an image of that person.

[0440] Next, the user starts a conversation with the system. When the user says, "I'm feeling very happy today," the device sends this speech to the emotion engine. The emotion engine recognizes the user's happy emotion and sends that information to the server. The server then generates a happy reaction in the other person's character and displays it to the user.

[0441] Example prompt sentence:

[0442] Users provide a photo and voice of the person they want to reunite with, and the system generates a portrait of that person. When the user says, "I'm feeling very happy today," the system recognizes the user's emotion and generates a happy reaction in the portrait of the person they want to reunite with.

[0443] In this way, the system can tailor its interactions with the other persona depending on the user's emotions.

[0444] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0445] Step 1:

[0446] User provides information

[0447] Users provide photos, videos, audio, and text information about the person they want to reunite with. For example, they can take a photo of the person with their smartphone camera and record a voice message with the microphone. This information is used as input.

[0448] Step 2:

[0449] The device collects information

[0450] The device collects photos and audio provided by the user through the application. Specifically, it sends the photo data taken by the user and the audio data recorded by the user to the server. The input is the information provided by the user, and the output is the data sent to the server.

[0451] Step 3:

[0452] The server stores the information

[0453] The server receives the information sent from the device and stores it in a database. For example, photo data is stored in an image folder and audio data is stored in an audio folder. The input is the data sent from the device, and the output is the information stored in the database.

[0454] Step 4:

[0455] The server inputs information into the AI ​​model

[0456] The server inputs the stored information into an AI model. Specifically, photo data and audio data are input into an AI model (such as GPT-4 or DALL-E) to learn the other person's appearance and speaking style. The input is the information stored in the database, and the output is the learning result of the AI ​​model.

[0457] Step 5:

[0458] AI model generates a portrait of the other person

[0459] The AI ​​model generates a portrait of the other person based on the input information. Specifically, it extracts the other person's facial features from a photo and learns speaking characteristics from audio. The input is data provided by the server, and the output is the generated portrait of the other person.

[0460] Step 6:

[0461] The device collects the user's emotions

[0462] The device collects the user's voice and facial expressions through a camera and microphone. For example, the microphone records the user's speech and the camera captures their facial expressions. The input is the user's voice and facial expressions, and the output is the collected emotional data.

[0463] Step 7:

[0464] Emotion engine analyzes emotions

[0465] The emotion engine analyzes collected voice and facial expression data to recognize the user's emotions. Specifically, it determines whether the user is happy or sad based on changes in voice tone and facial expression. The input is the collected emotional data, and the output is the analyzed emotional information.

[0466] Step 8:

[0467] The server generates a response based on the emotion.

[0468] The server adjusts the reaction of the other person's figure based on the emotional information received from the emotion engine. For example, if the user expresses joy, the server makes the other person's figure generate a joyful reaction. The input is the analyzed emotional information, and the output is the adjusted reaction of the other person.

[0469] Step 9:

[0470] The device displays the response to the user

[0471] The device displays the reaction of the other person's image received from the server to the user. For example, the other person's image is displayed with a smiling face on the screen of a smartphone. The input is the reaction data sent from the server, and the output is the other person's image displayed to the user.

[0472] (Application example 1)

[0473] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0474] Conventional virtual shopping experiences have struggled to provide personalized support to users, particularly in terms of responding to the user's emotions. Furthermore, there is a lack of technology to realistically recreate the person the user wants to meet and adjust their response based on the user's emotions.

[0475] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for recognizing the user's emotions and adjusting the reaction of the personality of the person in accordance with the emotions, and means for functioning as a shopping assistant in a virtual store. This makes it possible to provide the user with a personalized shopping experience and respond according to the user's emotions.

[0476] "Profile of the person you wish to meet again" refers to information including characteristics such as appearance, speaking style, personality, and preferences of a specific person that the user wishes to meet again.

[0477] "Means of understanding" refers to technology that uses artificial intelligence to analyze and learn about the other person's personality based on information provided by the user (photos, videos, audio, text information, etc.).

[0478] "Means of recreating realistic interactions" refers to technology that uses a generated image of the other person to mimic actual conversations and communication with the user.

[0479] "Means of visual reproduction" refers to technology that uses image recognition technology and computer graphics technology to realistically reproduce the appearance of the other person.

[0480] "A means to meet future partners" refers to technology that not only uses past information, but also generates and reproduces the image of a person you would like to meet again in the future.

[0481] "Means of recognizing emotions" refers to technology that analyzes emotions from the user's voice and facial expressions and understands those emotions.

[0482] "Means for adjusting the reaction of the other person's character according to their emotions" refers to a technology that changes the reaction of the generated other person's character based on the user's recognized emotions.

[0483] "Virtual store" refers to a virtual store operated on the Internet, which provides an environment where users can browse and purchase products online.

[0484] "Means to function as a shopping assistant" refers to technology in which a generated persona provides information and advice about products to support the user's shopping experience.

[0485] The system for implementing this invention has the function of understanding the profile of the person the user wants to reunite with and adjusting the response of that profile according to the user's emotions. A specific embodiment of this system will be described below.

[0486] System Configuration

[0487] The system consists of the following major components:

[0488] 1. User terminal: A device such as a smartphone or head-mounted display.

[0489] 2. Server: A central server that hosts AI models and databases.

[0490] 3. Camera: A camera to capture the user's facial expressions and movements.

[0491] 4. Microphone: A microphone to capture the user's voice.

[0492] Hardware and software used

[0493] Hardware: smartphone, head-mounted display, camera, microphone.

[0494] Software: OpenCV (acquisition and display of camera images), EmotionRecognizer (library for emotion recognition), AI PersonaGenerator (library for generating person images using AI).

[0495] Data processing and calculation

[0496] 1. User Device:

[0497] Collect information provided by users (photos, videos, audio, text information).

[0498] It uses a camera and microphone to capture the user's facial expressions and voice in real time.

[0499] 2. Server:

[0500] Based on the collected user information, an AI model is used to generate a profile of the person the user would like to meet again.

[0501] EmotionRecognizer is used to recognize emotions from captured user facial expressions and voice.

[0502] Based on the recognized emotions, the AI ​​PersonaGenerator is used to adjust the reactions of the other person's character.

[0503] Specific examples

[0504] While users are shopping in a virtual store, the image of the person they want to meet will happily say, "This product is perfect for you!" If the user is tired, the image will gently say, "Shall we take a break?"

[0505] Prompt Sentence Examples

[0506] Generate a persona of the person you want to reunite with based on photos, videos, audio, and text information provided by the user. The generated persona should adjust its reaction according to the user's emotions. If the user is happy, show a happy reaction, and if the user is sad, show a comforting reaction.

[0507] This system allows users to enjoy a more personalized shopping experience in a virtual store, and responds to users' emotions, making interactions more natural and friendly.

[0508] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0509] Step 1:

[0510] The user device collects information provided by the user (photos, videos, audio, and text information). This information becomes input data for generating a profile of the person the user wants to meet. Specifically, the user uses a smartphone or head-mounted display to take photos, record videos and audio, and input text information.

[0511] Step 2:

[0512] The user terminal sends the collected information to the server, which receives the information and stores it in a database for subsequent processing.

[0513] Step 3:

[0514] Based on the received information, the server uses a generative AI model to generate a portrait of the person the user wants to meet. Specifically, it uses the AI ​​PersonaGenerator library to analyze and integrate appearance from photos and videos, speaking style from audio, and personality and preferences from text information. The input for this process is the information provided by the user, and the output is a generated portrait.

[0515] Step 4:

[0516] The user device uses a camera and microphone to capture the user's facial expressions and voice in real time, and the captured data becomes input data for recognizing the user's emotions.

[0517] Step 5:

[0518] The user device sends the captured facial and voice data to the server, which receives the data and analyzes the user's emotions using the EmotionRecognizer library. The input of this process is the captured facial and voice data, and the output is the recognized emotion.

[0519] Step 6:

[0520] The server adjusts the reaction of the generated character based on the recognized emotion. Specifically, it uses the AI ​​PersonaGenerator library to generate a reaction according to the user's emotion. For example, if the user is happy, the character will show a happy reaction, and if the user is sad, the character will show a comforting reaction. The input of this process is the recognized emotion, and the output is the adjusted reaction.

[0521] Step 7:

[0522] The user device displays the adjusted response to the user. Specifically, the generated character is displayed on the screen of a smartphone or head-mounted display, speaking to the user. The input of this process is the adjusted response, and the output is the video and audio displayed to the user.

[0523] These steps allow users to enjoy a more personalized shopping experience in virtual stores, and also enable responses based on users' emotions, making interactions more natural and friendly.

[0524] Example 2

[0525] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0526] Conventional systems have difficulty realistically recreating the image of the person you want to reunite with, especially predicting and displaying their future appearance. Furthermore, they lacked the functionality to adjust the other person's future appearance based on the user's emotions, limiting the user experience. Furthermore, the technology required for realistic dialogue was insufficient, making it difficult to achieve natural interactions with users.

[0527] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0528] In this invention, the server includes means for receiving user input, means for transmitting input data to the server, means for generating a response using a generative AI model, means for analyzing the user's emotions using an emotion engine and adjusting the other person's future appearance, means for transmitting the response data to the user terminal, and means for displaying the response to the user. This makes it possible to adjust the future appearance according to the user's emotions and to reproduce realistic dialogue.

[0529] A "means for receiving user input" is an interface through which a user provides information to the system in the form of text, voice, or other information.

[0530] "Means for sending input data to the server" refers to a communication means for transferring information provided by the user to the server.

[0531] A "means for generating responses using a generative AI model" is a means for using a pre-trained artificial intelligence model to generate appropriate responses based on user input.

[0532] "Means of using an emotion engine to analyze the user's emotions and adjust the other person's future appearance" is a technology that analyzes the user's emotional state and appropriately changes the other person's future appearance based on the results.

[0533] The "means for transmitting response data to a user terminal" refers to a communication means for transferring the generated response to the user's device.

[0534] A "means for displaying a response to a user" is an interface for visually or audibly displaying the generated response on the user's device.

[0535] "Means of recreating realistic interactions" refers to technology that mimics the speaking style and tone of the other person to achieve natural dialogue with the user.

[0536] "Visual reproduction means" refers to a technique that uses image recognition technology and computer graphics technology to realistically reproduce the appearance of the other person.

[0537] "A means to meet not only past but also future partners" is a technology that allows users to visually see not only past partners but also future partners.

[0538] The present invention provides a system that realistically recreates the image of a person with whom a user wishes to reunite, allowing the user to meet not only past but also future contacts. A specific embodiment of this system will be described below.

[0539] Hardware and software used

[0540] Hardware: High-performance servers, user devices (PCs, smartphones)

[0541] Software: Generative AI models (e.g., GPT-4), emotion engines, dialogue management systems

[0542] System Overview

[0543] The system receives user input, sends it to a server, and generates a response using a generative AI model. It then analyzes the user's emotions using an emotion engine and adjusts the other person's future appearance. The generated response data is sent to the user's device and displayed to the user.

[0544] Specific examples

[0545] 1. User Input

[0546] A user opens the app on their smartphone, types "Good morning" into the text box, and presses the send button.

[0547] 2. Sending input data

[0548] The terminal encrypts the text data "Good morning" and sends it to a server via the Internet.

[0549] 3. Response generation using generative AI models

[0550] The server passes the received data "Good morning" to the generative AI model, and the model generates a text response saying "Good morning."

[0551] 4. Emotion analysis using an emotion engine

[0552] The server then passes the generated "Good morning" response to the emotion engine, which analyzes the user's emotions. For example, if a user inputs "I want to meet my partner in 10 years' time," expressing a feeling of anticipation, the emotion engine will analyze that emotion and adjust the partner's future appearance.

[0553] 5. Sending response data

[0554] The server encrypts the generated response data "Good morning" and sends it to the user's terminal.

[0555] 6. Displaying a response to the user

[0556] The device decodes the received response data, "Good morning," and displays it on the screen. If the user instructs, "I want to meet my partner in 10 years," the device will display the partner in 10 years' time as a successful and happy person.

[0557] Prompt Sentence Examples

[0558] "If a user says 'Good morning,' please reply 'Good morning' in the other person's voice."

[0559] "If a user indicates, 'I want to see my partner in 10 years,' and expresses an emotion of anticipation, show them as successful and happy in 10 years."

[0560] In this way, the server, terminal, and user each play their respective roles, creating a system that enables real dialogue and coordination of the future.

[0561] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0562] Step 1:

[0563] The user uses the device to enter text.

[0564] Specifically, the user opens the app on their smartphone, types "Good morning" into the text box, and presses the send button.

[0565] Input: User text input "Good morning"

[0566] Output: Text data entered into the terminal

[0567] Step 2:

[0568] The terminal transmits the user's input data to the server.

[0569] Specifically, the terminal encrypts the text data "Good morning" and sends it to a server via the Internet.

[0570] Input: Text data entered into the terminal: "Good morning"

[0571] Output: Encrypted text data sent to the server

[0572] Step 3:

[0573] The server passes the received input data to the generative AI model and generates a response.

[0574] Specifically, the server passes the data "Good morning" to a generative AI model (e.g., GPT-4), and the model generates a text response saying "Good morning."

[0575] Input: Encrypted text data "Good morning" sent to the server

[0576] Output: The response text "Good morning" generated by the generative AI model

[0577] Step 4:

[0578] The server passes the generated response to the emotion engine to analyze the user's emotion.

[0579] Specifically, the server passes the generated response "Good morning" to the emotion engine, which then analyzes the user's emotions. For example, if a user inputs "I want to meet my partner in 10 years' time," expressing a feeling of anticipation, the emotion engine will analyze that emotion and adjust the partner's future appearance.

[0580] Input: The response text "Good morning" generated by the generative AI model

[0581] Output: User emotion data analyzed by the emotion engine

[0582] Step 5:

[0583] The server transmits the generated response data to the user terminal.

[0584] Specifically, the server encrypts the generated response data "Good morning" and sends it to the user's terminal.

[0585] Input: User emotion data analyzed by the emotion engine and the response text "Good morning" generated by the generative AI model

[0586] Output: Encrypted response data sent to the user's device

[0587] Step 6:

[0588] The terminal displays the received response data to the user.

[0589] Specifically, the device decodes the received response data, "Good morning," and displays it on the screen. If the user instructs, "I want to meet my partner in 10 years," the device will display the partner in 10 years' time as a successful and happy person.

[0590] Input: Encrypted response data sent to the user terminal

[0591] Output: The response text "Good morning" displayed to the user and the person's future appearance

[0592] (Application example 2)

[0593] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0594] Conventional virtual stores have faced the challenge of monotonous dialogue with users and a lack of flexibility to respond to their emotions and expectations. Furthermore, they lacked the means to help users imagine future usage scenarios, making it difficult to stimulate purchasing motivation.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0596] In this invention, the server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting the person not only in the past but also in the future, means for recognizing the user's emotions, means for adjusting the person's future appearance according to the user's emotions, means for introducing products in a virtual store, means for generating responses based on the user's input, and means for displaying the generated responses. This enables flexible responses according to the user's emotions and expectations, and by specifically depicting future usage scenarios, it is possible to increase the user's desire to purchase.

[0597] "Means to understand the personality of the person you wish to reunite with" refers to technology that allows users to collect and analyze the characteristics and information of the person they wish to reunite with.

[0598] "Means of recreating real interactions" refers to technology that allows natural communication, like an actual conversation, based on collected personal profiles.

[0599] "Visual reproduction means" refers to techniques that use image recognition technology and computer graphics technology to realistically reproduce the appearance of a target person.

[0600] "A means to meet not only past but also future counterparts" is a technology that allows users to visually see not only past but also future images of people.

[0601] "Means for recognizing user emotions" refers to technology for analyzing and recognizing emotions from a user's facial expressions and voice.

[0602] "Means for adjusting the future appearance of the other person according to the user's emotions" is a technology for dynamically changing the future appearance of the other person based on the user's emotional state.

[0603] "Means for introducing products in a virtual store" refers to technology for providing and explaining product information to users in a virtual environment.

[0604] "Means for generating a response based on user input" refers to technology for generating an appropriate response to a question or request from a user.

[0605] A "means for displaying the generated response" is a technique for visually or audibly presenting the generated response to the user.

[0606] A system for carrying out this invention has the following configuration: A server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting the person not only in the past but also in the future, means for recognizing the user's emotions, means for adjusting the person's future appearance according to the user's emotions, means for introducing products in a virtual store, means for generating a response based on a user's input, and means for displaying the generated response.

[0607] Program processing explanation

[0608] The server first collects and analyzes information provided by the user to understand the person the user wants to reunite with. This information includes the other person's appearance, voice, speaking style, and past events. Next, based on the collected information, it uses voice recognition technology and computer graphics technology to recreate a realistic interaction. This makes it possible to mimic the other person's speaking style and tone and visually reproduce them.

[0609] Furthermore, to recognize the user's emotions, the system uses a camera and microphone to analyze the user's facial expressions and voice. Using emotion recognition technology, the system grasps the user's emotional state in real time. Based on this emotional information, the system adjusts the future image of the other person. For example, if the user expresses feelings of expectation, the system will display the other person's future image as successful and happy.

[0610] In the virtual store, when a user asks a question about a product, the server uses a generative AI model to generate an appropriate response. Based on the user's input, the server generates a prompt and uses the OpenAI API to generate a response. The generated response is presented to the user visually or audibly.

[0611] Specific examples

[0612] For example, if a user asks, "Tell me about this product," the server will explain, "This product uses the latest technology and is very high-performance." If a user expresses anticipation by asking, "What will happen if I wear this outfit?", the server will describe a future scene, saying, "When you wear this outfit, you will look very attractive and will be praised by those around you."

[0613] Prompt Sentence Examples

[0614] A user asks, "Tell me about this product." The user emotion is "Interested."

[0615] In this way, it becomes possible to respond flexibly to the user's emotions and expectations, and by specifically describing future usage scenarios, it is possible to increase the user's desire to purchase.

[0616] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0617] Step 1:

[0618] The server collects and analyzes information provided by users to understand the person they wish to reunite with. As input, it receives information provided by users about the other person's appearance, voice, speaking style, and past events. This information is stored in a database and an analysis algorithm is used to extract the other person's characteristics. As output, it obtains the other person's characteristic data.

[0619] Step 2:

[0620] The server uses voice recognition and computer graphics technologies to recreate realistic interactions based on the collected feature data of the other party. As input, it uses the feature data of the other party obtained in step 1. It uses voice recognition technology to imitate the other party's speaking style and tone, and computer graphics technology to recreate the other party's appearance. As output, it generates a model of the other party with which it can have realistic interactions.

[0621] Step 3:

[0622] The server uses a camera and microphone to analyze the user's facial expressions and voice to recognize their emotions. It receives camera footage and audio data as input. It uses emotion recognition technology to analyze the user's emotional state in real time. The output is the user's emotional data.

[0623] Step 4:

[0624] The server adjusts the other person's future appearance based on the user's emotional data. As input, it uses the user's emotional data obtained in step 3 and the other person's model generated in step 2. If the user shows an expectant emotion, it adjusts the other person's future appearance as successful and happy. As output, it obtains the adjusted future model of the other person.

[0625] Step 5:

[0626] When a user asks a question about a product in the virtual store, the server uses a generative AI model to generate an appropriate response. As input, it receives the user's question and the user's emotion data obtained in step 3. It generates a prompt sentence and generates a response using the OpenAI API. As output, it obtains the generated response.

[0627] Step 6:

[0628] The server presents the generated response to the user visually or audibly. It uses the generated response obtained in step 5 as input. It presents the response to the user using a display or speaker. As output, the user can confirm the response.

[0629] In this way, it becomes possible to respond flexibly to the user's emotions and expectations, and by specifically describing future usage scenarios, it is possible to increase the user's desire to purchase.

[0630] Example 3

[0631] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0632] With current technology, it is difficult to realistically recreate the appearance and speaking style of a person in response to a user's request to meet a past or future partner. It is also difficult to adjust the other person's appearance in response to the user's emotions. Furthermore, the process of generating the other person's appearance data using a generative AI model and recreating it using CG technology is complex and requires efficient execution.

[0633] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting not only past but also future people, means for adjusting the appearance of the person in response to the user's emotions, means for generating appearance data of the person using a generative AI model, means for generating a prompt message and sending it to the generative AI model, and means for sending the generated appearance data to CG generation software to recreate the appearance of the person. This allows the user to have the experience of meeting a person from the past or future.

[0634] "Means for understanding the personality of the person you wish to reunite with" refers to technology that allows users to analyze the characteristics and information of the person they wish to reunite with and obtain this data.

[0635] "Means for reproducing realistic interactions" refers to technology that uses acquired information about the other party to imitate the other party's speaking style and tone, and realistically reproduces a conversation with the user.

[0636] "Means of visual reproduction" refers to technology that uses image recognition technology and CG technology to reproduce the appearance of the other person and present it visually to the user.

[0637] "A means to meet not only past but also future partners" is a technology that adds or subtracts the other person's age and generates the appearance of the past or future partner.

[0638] The "means for adjusting the appearance of the other party according to the user's emotions" is a technology for analyzing the user's emotions and adjusting the appearance of the other party according to those emotions.

[0639] "Means for generating appearance data of another party using a generative AI model" refers to technology for generating data about the appearance of another party using a generative AI model.

[0640] "Means for generating a prompt sentence and sending it to a generative AI model" refers to technology for generating a prompt sentence based on a user's instructions and sending the prompt sentence to a generative AI model.

[0641] "Means of sending the generated appearance data to CG generation software and reproducing the other person's appearance" refers to a technology for sending appearance data obtained from a generative AI model to CG generation software and reproducing the other person's appearance based on that data.

[0642] This invention is a system that realistically recreates the appearance and speaking style of a person in response to a user's desire to meet a past or future partner. This system includes multiple means for understanding the personality of the person the user wishes to reunite with, recreating realistic interactions, and visually recreating the person.

[0643] Hardware and software used

[0644] Hardware: High-performance servers, user devices (PCs, smartphones, etc.)

[0645] Software: Generative AI models (e.g., GPT-4), CG generation software (e.g., Blender), emotion engines (e.g., Affectiva)

[0646] Specific operation of the system

[0647] 1. Receive user instructions

[0648] A user uses a terminal to input an instruction to meet a person in the past or future. For example, the user inputs "I want to meet the person in 10 years' time."

[0649] 2. The server receives the instruction

[0650] The server receives instructions from the user and analyzes their content. For example, it analyzes an instruction such as "I want to meet my partner in 10 years' time."

[0651] 3. The server generates a prompt

[0652] The server generates a prompt to send to the generative AI model based on the instructions. For example, it creates a prompt such as "Please generate what the other person will look like in 10 years' time."

[0653] 4. The server sends a prompt to the generative AI model

[0654] The server sends the generated prompt to the generative AI model. For example, it sends the prompt to GPT-4: "Generate what the other person will look like in 10 years."

[0655] 5. The generative AI model generates the other person's appearance data

[0656] The generative AI model generates appearance data based on the prompt, for example, what the person's facial features will be like in 10 years.

[0657] 6. The server sends the appearance data to the CG generation software

[0658] The server sends the appearance data received from the generative AI model to CG generation software, for example, Blender, to see what the other person will look like in 10 years.

[0659] 7. Generative computer graphics software recreates the appearance of your opponent.

[0660] CG generation software then uses the received data to recreate the other person's appearance. For example, Blender creates a 3D model of what the other person will look like in 10 years' time.

[0661] 8. The server uses the emotion engine to adjust the appearance (if necessary)

[0662] If the user shows a certain emotion, the server uses the emotion engine to adjust the appearance, for example, if the user shows nostalgia, the emotion engine analyzes the emotion and adjusts the appearance as a special memorable scene.

[0663] 9. The server displays the final look to the user

[0664] The server then displays the final generated appearance on the user's device, for example, displaying the partner's face 10 years from now on the user's smartphone.

[0665] Examples and prompts

[0666] Example 1: When the user says "I want to meet my partner in 10 years"

[0667] Prompt: "Generate what your partner will look like in 10 years."

[0668] Processing: The server sends the prompt to the generative AI model, and the CG generation software uses the generated data to recreate the other person's appearance 10 years in the future.

[0669] Example 2: The user indicates nostalgia by saying, "I want to meet someone from five years ago."

[0670] Prompt: "Generate what your partner looked like five years ago and reflect your feelings of nostalgia."

[0671] Processing: The server sends the prompt text to the generative AI model, and based on the generated data, the CG generation software recreates the appearance of the other person five years ago, and the emotion engine adjusts a special memorable scene that reflects the emotion of nostalgia.

[0672] In this way, the user can have the experience of meeting a partner from the past or future. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0673] Step 1:

[0674] The user enters instructions

[0675] The user uses the terminal to input instructions about wanting to meet a partner in the past or future. For example, they can input "I want to meet the partner in 10 years' time" by voice. The input instructions are sent to the server as text data.

[0676] Step 2:

[0677] The server receives the instruction

[0678] The server receives instructions from the user and analyzes their content. For example, it analyzes the instruction "I want to meet the person I'll be with in 10 years" and understands the instruction content. The input is the user's instruction text, and the output is the analyzed instruction content.

[0679] Step 3:

[0680] The server generates a prompt

[0681] The server generates a prompt to send to the generative AI model based on the instructions. For example, it creates a prompt such as "Please generate what the other person will look like in 10 years." The input is the parsed instructions, and the output is the generated prompt.

[0682] Step 4:

[0683] The server sends a prompt to the generative AI model

[0684] The server sends the generated prompt to the generative AI model. For example, it sends the prompt to the generative AI model, "Please generate what the other person will look like in 10 years." The input is the generated prompt, and the output is the result sent to the generative AI model.

[0685] Step 5:

[0686] A generative AI model generates the other person's appearance data

[0687] The generative AI model generates appearance data based on the prompt. For example, it generates the facial features of the other person 10 years from now. The input is the prompt, and the output is the generated appearance data.

[0688] Step 6:

[0689] The server sends the appearance data to the CG generation software.

[0690] The server sends the appearance data received from the generative AI model to the CG generation software. For example, it sends data on the other person's face 10 years from now to the CG generation software. The input is the generated appearance data, and the output is the result sent to the CG generation software.

[0691] Step 7:

[0692] CG generation software recreates the other person's appearance

[0693] The CG generation software recreates the other person's appearance based on the received data. For example, the CG generation software recreates and renders a 3D model of what the other person's face will look like 10 years from now. The input is appearance data, and the output is the recreated 3D model.

[0694] Step 8:

[0695] The server uses the emotion engine to adjust the appearance (if necessary)

[0696] If the user shows a specific emotion, the server uses the emotion engine to adjust the appearance. For example, if the user shows nostalgia, the emotion engine analyzes the emotion and adjusts the appearance to represent a special memorable scene. The input is the user's emotion data, and the output is the adjusted appearance data.

[0697] Step 9:

[0698] The server displays the final look to the user

[0699] The server displays the final generated appearance on the user's device. For example, it displays the other person's face 10 years from now on the user's smartphone. The input is the final appearance data, and the output is the display result on the user's device.

[0700] (Application example 3)

[0701] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0702] Conventional virtual store systems did not allow users to interact with store clerks or customers from the past or future, nor could they adjust the appearance of the other person based on the user's emotions. This limited the user experience and left them with low emotional satisfaction. Furthermore, they were unable to provide the excitement and nostalgia that can be gained through interactions with past and future counterparts.

[0703] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0704] In this invention, the server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting the person not only in the past but also in the future, means for adjusting the past appearance of the person in accordance with the user's emotions, and means for interacting with past and future store clerks and customers in the virtual store. This allows the user to interact with past and future store clerks and customers, and by adjusting the appearance of the person in accordance with the user's emotions, it is possible to provide a more moving and nostalgic experience.

[0705] "Means to understand the personality of the person you wish to reunite with" refers to technology that recognizes the characteristics and information of a specific person with whom the user wishes to reunite and acquires this information as data.

[0706] "Means for reproducing realistic interactions based on the image of the other person" is a technology that uses acquired data on the image of the person to simulate natural conversation and communication with that person.

[0707] "Means for visually reproducing the other person" refers to a technology that uses image recognition technology or CG technology to visually reproduce the appearance of the person based on the acquired data of the person's image.

[0708] "A means to meet people not only from the past but also from the future" is a technology that uses AI technology to simulate past and future appearances, allowing users to interact with the person at that time.

[0709] "Means for adjusting the other person's past appearance according to the user's emotions" refers to a technology that recognizes the user's emotional state and adjusts the other person's past appearance accordingly to match a specific emotion.

[0710] "Means for interacting with past and future store clerks and customers in a virtual store" refers to technology that enables users to interact with past and future store clerks and customers in a virtual store.

[0711] The system for implementing this invention includes multiple means for understanding the personality of a person you want to meet again, recreating and visually recreating real interactions, as well as means for meeting past and future counterparts, means for adjusting the past appearance of counterparts according to the user's emotions, and means for interacting with past and future store clerks and customers in a virtual store.

[0712] Hardware and software used

[0713] Hardware: Smartphone, head-mounted display

[0714] Software: OpenCV, DeepFace, EmotionRecognizer, AgeTransformer

[0715] Data processing and calculation

[0716] 1. A way to understand the person you want to reunite with

[0717] The server receives photos and videos of the other person provided by the user and extracts their features using image recognition technology. Specifically, it analyzes the images using OpenCV and recognizes facial features using DeepFace.

[0718] 2. A way to recreate real-life interactions

[0719] The server uses voice recognition technology to imitate the speaking style and tone of the other person, allowing the user to experience a natural conversation. The voice data is analyzed using a voice recognition engine, and data is generated to reproduce the speaking style and tone of the other person.

[0720] 3. Visual Representation

[0721] The server uses CG technology to recreate the appearance of the person based on the acquired facial feature data. Specifically, based on the facial features recognized by DeepFace, it performs age transformation using AgeTransformer and visually recreates the appearance using OpenCV.

[0722] 4. A way to meet past and future partners

[0723] The server adds or subtracts the other person's age based on the user's instructions to generate future or past appearances. For example, if the user requests that they "want to meet the other person 10 years from now," the server uses AgeTransformer to change the other person's appearance 10 years from now.

[0724] 5. A way to adjust the past appearance of the other person depending on the user's emotions

[0725] The server uses EmotionRecognizer to recognize the user's emotions and adjusts the past appearance of the other person according to their emotions. For example, if the user expresses nostalgia, it applies a specific filter to recreate a nostalgic appearance.

[0726] 6. A way to interact with past and future store associates and customers in virtual stores

[0727] The server enables the user to interact with past and future store clerks and customers in the virtual store, allowing the user to enjoy natural interactions with store clerks and customers at a specific point in time in the virtual store.

[0728] Specific examples

[0729] If the user indicates "I want to meet the store clerk from 10 years ago" and expresses nostalgia, the server will display the store clerk's appearance from 10 years ago with a nostalgic filter applied.

[0730] Prompt Sentence Examples

[0731] If the user indicates "I want to meet the store clerk from 10 years ago" and expresses nostalgia, the system should display the store clerk as they were 10 years ago with a nostalgic filter applied.

[0732] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0733] Step 1:

[0734] The user uploads a photo of the person they want to meet again to their device.

[0735] Input: A user-provided photo of the person.

[0736] Operation: The device sends the photo file specified by the user to the server.

[0737] Output: The photo data of the other person sent to the server.

[0738] Step 2:

[0739] The server receives the photo and uses image recognition technology to extract the other person's features.

[0740] Input: The photo data of the other person sent to the server.

[0741] How it works: The server uses OpenCV to analyze the photo and DeepFace to recognize facial features.

[0742] Output: Extracted facial feature data.

[0743] Step 3:

[0744] The server converts the other person's age based on the user's instructions.

[0745] Input: extracted facial feature data, user instructions (e.g., "I want to meet the person I will be in 10 years' time").

[0746] How it works: The server applies an age transformation to facial feature data using the AgeTransformer, changing the appearance by the specified number of years.

[0747] Output: Age-transformed facial feature data.

[0748] Step 4:

[0749] The server recognizes the user's emotions and adjusts the other person's appearance accordingly.

[0750] Input: Age-transformed facial feature data, user emotion data.

[0751] How it works: The server uses EmotionRecognizer to analyze the user's emotions and applies filters to adjust the other person's appearance based on their emotions.

[0752] Output: Refined facial feature data.

[0753] Step 5:

[0754] Based on the adjusted facial feature data, the server uses CG technology to visually recreate the other person's appearance.

[0755] Input: Aligned facial feature data.

[0756] How it works: The server uses CG technology to visually recreate facial feature data and generate an image.

[0757] Output: Reconstructed image of the other person's appearance.

[0758] Step 6:

[0759] The server transmits the reproduced image of the other person's appearance to the terminal and displays it to the user.

[0760] Input: Reconstructed image of the other person's appearance.

[0761] How it works: The server sends the generated image to the device, which displays it to the user.

[0762] Output: An image of the other person's appearance displayed to the user.

[0763] Step 7:

[0764] The user interacts with the person reproduced in the virtual store.

[0765] Input: Appearance image of the other person displayed to the user, voice input.

[0766] How it works: The device sends the user's voice input to a server, which then uses voice recognition technology to mimic the speaking style and tone of the other person, recreating a realistic interaction.

[0767] Output: Interaction data between the user and the reproduced opponent.

[0768] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0769] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0770] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0771] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0772] [Second embodiment]

[0773] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0774] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0775] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0776] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0777] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0778] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0779] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0780] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0781] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0782] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0783] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0784] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0785] "Example 1"

[0786] In one embodiment of the present invention, information provided by the user (e.g., photos, videos, audio, text information, etc.) is collected as a means of understanding the personality of the person the user wishes to reunite with, and an image of the person is generated based on this information using artificial intelligence (AI). The AI ​​learns the person's appearance, speaking style, personality, preferences, etc., and generates an image of the person based on this information.

[0787] "Example 2"

[0788] In addition, as a means of recreating realistic interactions, the AI ​​will converse with the user by imitating the other person's speaking style and tone based on the generated persona. For example, if the user greets the user with "Good morning," the AI ​​will respond with "Good morning" in the other person's tone.

[0789] "Example 3"

[0790] Furthermore, as a means of visually recreating the other person, CG technology is used to mimic the other person's appearance based on the generated image. For example, based on a photo of the other person, AI can recreate their face using CG and display it to the user.

[0791] "Example 4"

[0792] To allow users to meet their partner not only in the past but also in the future, the AI ​​will add or subtract their age and change their appearance. For example, if a user requests to meet their partner in 10 years, the AI ​​will apply the changes to their appearance in 10 years' time and display the results to the user.

[0793] The processing flow of each embodiment will be described below.

[0794] "Example 1"

[0795] Step 1: Collect information from the user about the person they wish to reunite with (e.g., photos, videos, audio, text information, etc.).

[0796] Step 2: Using the collected information, artificial intelligence (AI) is used to generate a profile of the person. The AI ​​learns the person's appearance, speech pattern, personality, preferences, etc.

[0797] "Example 2"

[0798] Step 1: Based on the generated image, the AI ​​imitates the other person's speaking style and tone.

[0799] Step 2: Dialogue with the user. For example, if the user says "Good morning," the AI ​​will respond with "Good morning" in the user's voice.

[0800] "Example 3"

[0801] Step 1: Using the generated image, CG technology is used to mimic the appearance of the other person.

[0802] Step 2: For example, based on a photo of the other person, the AI ​​will recreate their face in CG and display it to the user.

[0803] "Example 4"

[0804] Step 1: The AI ​​adds or subtracts the opponent's age and changes their appearance.

[0805] Step 2: For example, if a user says, "I want to meet my partner 10 years from now," the AI ​​will apply changes to the partner's appearance in 10 years' time and display the results to the user.

[0806] Example 1

[0807] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0808] There is a demand for a system that can provide a detailed understanding of the personality of a person with whom a user wishes to reunite and allow the user to recreate a realistic interaction with that person. However, conventional technologies have had difficulty in comprehensively analyzing the other person's appearance, speech style, personality, preferences, etc. to generate an accurate personality profile. Furthermore, there has been a lack of means to effectively provide the generated personality profile to the user. As a result, users have been unable to obtain detailed information about the person they wish to reunite with, and have been unable to have a satisfying reunion experience.

[0809] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0810] In this invention, the server includes means for collecting information provided by users, means for analyzing the collected information, means for generating a person's profile based on the analyzed information, and means for providing the generated person's profile to the user, thereby enabling the user to accurately understand the detailed profile of the person they wish to reunite with and to recreate a realistic interaction.

[0811] The "means of collecting information provided by users" refers to a function that allows users to upload photos, videos, audio, text information, etc. related to the person they wish to reunite with to the system, which is then received and stored by the server.

[0812] "Means for analyzing collected information" refers to a function that analyzes the photos, videos, audio, and text information collected by the server and extracts characteristics such as the other person's appearance, speaking style, personality, and preferences.

[0813] "Image recognition technology" is a technology for analyzing the external characteristics of others from photographs and videos, and includes algorithms that detect face shape, hairstyle, clothing, etc.

[0814] "Speech recognition technology" is a technology that converts voice data into text and further analyzes the tone and speaking characteristics of the voice.

[0815] "Natural language processing technology" is a technology for analyzing text data and inferring the personality and preferences of others, and involves the use of generative AI models.

[0816] The "means for generating a person's profile" is a function that creates a detailed profile of the other person by integrating the other person's appearance, speaking style, personality, preferences, etc. based on analyzed information.

[0817] The "means for providing the generated portrait to the user" is a function for notifying the user of the generated portrait on the user's terminal and displaying the profile.

[0818] This invention is a system for enabling a user to understand in detail the personality of a person with whom they wish to reunite and to recreate realistic interactions. This system includes a series of processes that collect and analyze information provided by the user, generate a personality profile of the other person, and provide it to the user.

[0819] First, users upload photos, videos, audio, and text information about the person they want to reunite with to the system. Users can easily provide this information using a smartphone application. For example, users can take photos of the person they want to reunite with and record and upload audio messages.

[0820] The server then collects the information provided by the user and stores it in a database, specifically using cloud storage services such as AWS S3 buckets to store photos and audio files.

[0821] The collected information is analyzed by a server using image recognition technology, such as OpenCV and TensorFlow, to analyze photos and videos and extract the other person's external characteristics, such as facial shape, hairstyle, and clothing.

[0822] Using voice recognition technology, the server converts the voice data into text using voice recognition services such as Google Speech-to-Text API or IBM Watson, and then analyzes the voice tone and speaking characteristics.

[0823] Using natural language processing technology, the server uses generative AI models such as GPT-3 and BERT to analyze text extracted from written and audio data and infer the other person's personality and preferences.

[0824] Based on these analysis results, the server generates a profile of the other person, including their appearance, speaking style, personality, and preferences. For example, it creates a profile that says, "The other person has short hair, speaks softly, and enjoys the outdoors."

[0825] Finally, the server provides the generated profile to the user. Specifically, the server sends a notification to the user's device and displays the generated profile. The user can then open the application and check the detailed profile of the other person.

[0826] An example of a prompt is as follows:

[0827] "You provide a photo and an audio message of the person you want to reunite with. Based on this information, we analyze their appearance, speaking style, personality, and preferences to generate a profile of them."

[0828] In this way, the user can accurately understand the detailed personality of the person they wish to meet again and recreate a realistic interaction.

[0829] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0830] Step 1: User Provides Information

[0831] Users upload photos, videos, audio, text information, etc. of the person they want to reunite with to the system. Specifically, users use a smartphone application to take photos of the person they want to reunite with and record and upload audio messages. The input is the photos, videos, audio, and text information provided by the user, and the output is the transmission of this data to the server.

[0832] Step 2: The server collects information

[0833] The server receives the information provided by the user and stores it in a database. Specifically, the server uses cloud storage services such as AWS S3 buckets to store photos and audio files. The input is the photos, videos, audio, and text information sent by the user, and the output is that this data is stored in cloud storage.

[0834] Step 3: The server analyzes the image

[0835] The server uses image recognition libraries such as OpenCV and TensorFlow to analyze the stored photos and videos. Specifically, the server runs a face detection algorithm to extract the facial features of the other person (eye position, nose shape, mouth shape, etc.). The input is the photos and videos stored in cloud storage, and the output is the extracted facial feature data.

[0836] Step 4: The server analyzes the audio

[0837] The server uses speech recognition services such as Google Speech-to-Text API and IBM Watson to analyze the stored voice data. Specifically, the server converts the voice data into text and further extracts voice tone and speaking characteristics. The input is the voice data stored in cloud storage, and the output is text data and voice feature data.

[0838] Step 5: The server parses the text

[0839] The server uses natural language processing models such as GPT-3 and BERT to analyze the text information provided by the user. Specifically, the server analyzes the text data and infers the other person's personality and preferences. The input is the text information stored in cloud storage, and the output is the analyzed personality and preference data.

[0840] Step 6: The server integrates the information and generates a profile

[0841] The server integrates features extracted from images, voice, and text information to generate a profile of the other person. Specifically, the server compiles the other person's appearance, speaking style, personality, and preferences into a single profile. The input is the extracted facial feature data, voice feature data, personality and preference data, and the output is the generated profile of the person.

[0842] Step 7: The server provides the generated profile to the user

[0843] The server provides the generated persona to the user. Specifically, the server sends a notification to the user's device and displays the generated profile. The input is the generated persona profile, and the output is the profile displayed on the user's device.

[0844] (Application example 1)

[0845] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0846] In modern society, it is not easy for people to find someone they want to reunite with. This is especially true in large spaces such as large brick-and-mortar stores and shopping malls. It is also difficult to accurately reproduce the appearance, manner of speech, personality, and other characteristics of the person you want to reunite with and have a realistic interaction. This often leads to missed reunion opportunities, which leads to a decline in user satisfaction.

[0847] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0848] In this invention, the server includes means for understanding the personality of a person the user wishes to reunite with, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for searching for the person in a physical store, means for notifying the user when the person is found, and means for allowing the user to meet not only past but also future partners. This allows the user to easily find the person they wish to reunite with even in a large physical store, and allows for realistic interactions that accurately reproduce the person's characteristics.

[0849] "A means of understanding the personality of the person you want to reunite with" is a technology in which AI learns the other person's appearance, speaking style, personality, preferences, etc. based on information provided by the user (photos, videos, audio, text information, etc.), and generates an image of that person.

[0850] "Means for recreating realistic interactions" refers to technology that uses voice recognition technology and natural language processing technology to imitate the speaking style and tone of the other person based on a generated image of the other person, and engage in realistic conversations with the user.

[0851] "Means of visual reproduction" refers to technology that uses image recognition technology and computer graphics technology to imitate the appearance of the other person and visually reproduce the other person for the user.

[0852] The "means of finding a partner within a physical store" refers to a technology that analyzes camera footage within a physical store in real time and uses facial recognition technology to find the person you want to meet again with.

[0853] The "means for notifying when the person is found" refers to a technology that sends a notification to a device such as smart glasses or a smartphone when a person you want to reunite with is found, informing the user.

[0854] "A means to meet not only past but also future partners" is a technology that allows you to save information about not only past partners you want to meet again with, but also people you want to meet again with in the future, and then reunite with those people in the future.

[0855] The system for implementing this invention has the function of understanding the profile of the person the user wants to meet, searching for that person in a physical store, and notifying the user when the person is found. A specific embodiment of this system will be described below.

[0856] First, the user provides information about the person they want to reunite with. This information includes photos, videos, audio, and text of the person. This information is sent to a server and used as training data for an AI model. The server uses facial recognition libraries such as DeepFace and face_recognition to learn the person's appearance, speaking style, personality, preferences, etc., and generates a profile of that person.

[0857] Next, to find the person in the physical store, the server analyzes the camera footage in the physical store in real time. It uses image processing libraries such as OpenCV to detect faces from the camera footage and identifies the person using facial recognition technology. The server then matches the detected face with an AI model to find the person you want to reunite with.

[0858] When a match is found, the server sends a notification to the device, such as smart glasses or a smartphone. The notification informs the user that the match has been found and offers an opportunity to reunite. The notification is displayed on the smart glasses display or the smartphone's notification function.

[0859] For example, a user enters a photo and characteristics of a person they would like to meet in a shopping mall into the smart glasses. The server trains an AI model based on the provided information and analyzes camera footage in the mall to find the person. When the person is found, the smart glasses display a notification saying "Partner Found!"

[0860] Example prompt sentence:

[0861] Develop an application that allows users to provide photos, videos, audio, and text information about the person they wish to reunite with, and uses AI to generate a profile of that person. The application will be installed on smart glasses and will help users find their match in a physical store. Specifically, the AI ​​will generate a profile of the person based on the information provided by the user, and then locate that person in the store in real time.

[0862] In this way, users can easily find the person they want to meet again even in a large physical store, and can have a realistic interaction that accurately reproduces the other person's characteristics.

[0863] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0864] Step 1:

[0865] The user provides information about the person they want to meet. The user inputs photos, videos, audio, and text information about the other person into their device. This information is then sent to the server. The input data includes characteristics such as the other person's appearance, speaking style, personality, and preferences.

[0866] Step 2:

[0867] The server trains an AI model based on the information provided. The server uses facial recognition libraries such as DeepFace and face_recognition to learn the other person's appearance. It also uses voice recognition technology to learn the other person's speaking style and tone. This generates a profile of the other person. The input data is the information provided by the user, and the output data is the trained AI model.

[0868] Step 3:

[0869] The server analyzes camera footage from inside the physical store in real time. The server uses image processing libraries such as OpenCV to detect faces from the camera footage. The detected faces are compared with an AI model. The input data is the camera footage from inside the physical store, and the output data is the location information of the detected faces.

[0870] Step 4:

[0871] The server uses facial recognition technology to identify the person you want to reunite with. The server then matches the detected face with an AI model to find the person. The input data is the location information of the detected face, and the output data is the result of identifying the person.

[0872] Step 5:

[0873] The server sends a notification when the other party is found. The server then sends a notification to a device such as smart glasses or a smartphone to notify the user. The notification includes a message indicating that the other party has been found. The input data is the result of identifying the other party, and the output data is the notification message.

[0874] Step 6:

[0875] The user receives a notification and has the opportunity to reunite. The user checks the notification displayed on the smart glasses or smartphone and takes action to reunite with the other person. The input data is the notification message, and the output data is the user's action.

[0876] Example 2

[0877] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0878] With current communication technology, it is difficult to realistically recreate the image of someone you want to meet again. Furthermore, there is a lack of means to interact realistically with not only past contacts but also future contacts. Furthermore, technology to generate natural dialogue based on user input is not yet fully developed. To solve these issues, a system is needed that can mimic the speaking style and tone of the other person and visually recreate them.

[0879] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0880] In this invention, the server includes means for understanding the personality of a person the user wishes to reunite with, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting not only past but also future people, means for receiving input from the user, means for converting the received input into a prompt sentence, means for generating a response from the prompt sentence using a generative AI model, and means for displaying the generated response to the user, thereby enabling the user to have a realistic conversation with past or future people.

[0881] "Means to understand the personality of the person you would like to meet again" refers to technology for collecting and analyzing data such as the characteristics, personality, and speaking style of people you have met in the past or people you would like to meet in the future.

[0882] "Means of recreating real interactions" refers to technology that mimics the speaking style and tone of the other person based on collected personalities, allowing for natural dialogue with the user.

[0883] "Means of visual reproduction" refers to techniques that use image recognition technology and computer graphics technology to realistically reproduce the appearance and facial expressions of the other person.

[0884] "A means to meet people not only from the past but also in the future" is a technology that can predict and reproduce the characteristics of not only people you have met in the past, but also people you would like to meet in the future.

[0885] The "means for receiving input from the user" is an interface for sending text or voice input by the user to the server.

[0886] "Means for converting received input into prompt text" refers to technology for converting text or speech received from a user into a format that is easy for a generative AI model to understand.

[0887] "Means for generating a response from a prompt sentence using a generative AI model" refers to technology for using a generative AI model to generate an appropriate response based on a prompt sentence.

[0888] "Means for displaying the generated response to the user" means an interface for visually or audibly presenting the response obtained from the generative AI model to the user.

[0889] This invention is a system that enables users to have realistic conversations with people from the past or future. This system operates among three parties: a server, a terminal, and a user.

[0890] Hardware and software used

[0891] Hardware:

[0892] A server equipped with a high-performance GPU (e.g., NVIDIA Tesla V100)

[0893] software:

[0894] Generative AI models (e.g., GPT-4)

[0895] System Operation Overview

[0896] server:

[0897] The server receives input from the user, converts it into a prompt, uses a generative AI model to generate a response from the prompt, and sends the response to the device for display to the user.

[0898] Device:

[0899] The terminal transmits the text entered by the user to the server and displays the response received from the server to the user.

[0900] user:

[0901] The user enters text into the terminal and checks the response from the server.

[0902] Specific examples

[0903] Take the example of a user typing "good morning" into a terminal.

[0904] 1. Receiving user input:

[0905] The user types "Good morning" into the chat application on their device.

[0906] The terminal transmits the entered text to the server in real time.

[0907] 2. Generate prompt sentence:

[0908] Based on the received text "Good morning," the server generates a prompt sentence: "When the user says 'Good morning,' please reply in the other person's tone of voice."

[0909] 3. Generative AI model generates responses:

[0910] The server inputs the generated prompt sentence into the generative AI model, which analyzes the prompt sentence and generates the appropriate response, "Good morning."

[0911] 4. Sending a response:

[0912] The server sends the response "Good morning" obtained from the generative AI model to the device.

[0913] 5. User Visibility:

[0914] The device receives the reply "Good morning" from the server and displays it on the chat application screen, allowing the user to see the reply imitating the other person's speaking style and tone.

[0915] In this way, users can have realistic conversations with people from the past or future. The system uses generative AI models to create natural conversations based on user input.

[0916] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0917] Step 1: Receiving User Input

[0918] User: Type text into the device's chat application (e.g. "Good morning").

[0919] Terminal: Receives text entered by the user and sends it to the server.

[0920] Input: The text entered by the user (e.g., "Good morning").

[0921] Output: The text data sent to the server.

[0922] Step 2: Generate a prompt statement

[0923] Server: Generates a prompt based on the received user text.

[0924] Specific operation: The server analyzes the received text "Good morning" and generates a prompt sentence: "When the user says 'Good morning,' please reply in the other person's tone of voice."

[0925] Input: The text received from the user (e.g. "Good morning").

[0926] Output: The generated prompt (e.g., "When the user says 'Good morning,' please respond in the other person's voice tone.").

[0927] Step 3: Generative AI model generates a response

[0928] Server: Inputs the generated prompt sentence into the generative AI model and generates a response.

[0929] Specific operation: The server inputs a prompt sentence into a generative AI model (e.g., GPT-4), which analyzes it and generates the appropriate response, "Good morning."

[0930] Input: The generated prompt (e.g., "When the user says 'Good morning,' please respond in their voice.").

[0931] Output: The response obtained from the generative AI model (e.g., "Good morning").

[0932] Step 4: Send your response

[0933] Server: Sends the answers obtained from the generative AI model to the device.

[0934] Specific operation: The server sends the generated response "Good morning" to the terminal.

[0935] Input: A response obtained from a generative AI model (e.g., "Good morning").

[0936] Output: The response data sent to the device.

[0937] Step 5: Display to the user

[0938] Terminal: Displays the response received from the server to the user.

[0939] Specific operation: The device displays the reply "Good morning" received from the server on the chat application screen.

[0940] Input: Response data received from the server (e.g. "Good morning").

[0941] Output: The response that is displayed to the user (e.g. "Good morning").

[0942] In this way, users can experience realistic interactions.

[0943] (Application example 2)

[0944] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0945] In conventional customer service, it was difficult to provide personalized service based on customer preferences and past purchase history, making it difficult to improve customer satisfaction. In addition, the way a salesperson speaks and their tone of voice have a significant impact on customers, so there was a need for a way to improve the customer experience by imitating the speaking style and tone of a specific salesperson.

[0946] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for understanding the personality of a person the customer wants to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting not only past but also future people, means for providing customer service by imitating the customer's speaking style and tone, means for supporting dialogue with the customer using smart glasses, and means for imitating the customer's speaking style and tone using a generative AI model. This enables personalized customer service based on the customer's preferences and past purchase history, and makes it possible to improve the customer experience by imitating the speaking style and tone of a specific store clerk.

[0947] "A means of understanding the personality of the person you wish to reunite with" is a technology that analyzes the characteristics, personality, and past interactions of a specific person with whom the user wishes to reunite, and saves this data.

[0948] "Means of recreating realistic interactions" refers to technology that uses stored character data to mimic actual conversations and behaviors, enabling natural communication with users.

[0949] "Means of visual reproduction" refers to techniques that use image recognition technology and CG technology to realistically reproduce the appearance and facial expressions of a specific person.

[0950] "A means to meet people not only from the past but also from the future" is a technology that generates images of people who may exist in the future as well as people who existed in the past, allowing users to interact with them.

[0951] "Means of serving customers by imitating the way a customer speaks and the tone of their speech" refers to a technique of analyzing the way a customer speaks and the tone of their speech and imitating it to serve customers.

[0952] "Means for supporting conversations with customers using smart glasses" refers to technology that allows store clerks wearing smart glasses to support conversations with customers.

[0953] "Means of using a generative AI model to mimic a customer's speaking style and tone" refers to a technology that uses a generative AI model to mimic a customer's speaking style and tone, enabling natural dialogue.

[0954] A system for implementing this invention includes means for understanding the personality of a person you wish to reunite with, means for recreating real interactions, means for visually recreating interactions, means for meeting not only past but also future counterparts, means for serving customers by imitating the speaking style and tone of voice of the customer, means for supporting interactions with customers using smart glasses, and means for imitating the speaking style and tone of voice of the customer using a generative AI model.

[0955] Hardware and software used

[0956] Hardware: Smart glasses (with built-in camera and microphone)

[0957] Software: OpenAI API, OpenCV, pyttsx3

[0958] System Operation

[0959] The server first analyzes the information provided by the user to understand the person they want to reunite with. This information includes specific person characteristics and past interactions. The analyzed data is then stored using a generative AI model.

[0960] To replicate realistic interactions, a generative AI model is then used to generate natural-looking dialogue based on the stored persona data, allowing users to interact with a specific person in a realistic way.

[0961] The visual reproduction is achieved by using image recognition and CG technology to realistically reproduce the appearance and facial expressions of a specific person, allowing users to visually feel as if they are interacting with that person.

[0962] Furthermore, as a means to meet people not only from the past but also from the future, the technology generates images of people who may exist in the future and allows users to interact with them. This technology is realized using a generative AI model.

[0963] As a means of serving customers by imitating their speaking style and tone, store clerks wearing smart glasses can analyze the speaking style and tone of the customer and imitate them when serving them, allowing customers to receive more personalized service.

[0964] The smart glasses are used to support customer interactions by capturing the customer's voice input in real time using the glasses' camera and microphone, and then using a generative AI model to generate responses that mimic the customer's speaking style and tone. The generated responses are then played back aloud using pyttsx3.

[0965] Specific examples

[0966] For example, if a customer says "Good morning," the smart glasses' microphone captures the audio and sends it to the generative AI model, which then uses the following prompt sentence to generate a response that mimics the customer's speaking style and tone:

[0967] Example prompt sentence:

[0968] Respond by imitating the customer's speaking style and tone: Good morning

[0969] The generated response is played back aloud using pyttsx3 in the form of "Good morning, what product are you looking for today?" This allows customers to experience a natural dialogue that mimics the speaking style and tone of a specific store clerk.

[0970] In this way, a system for implementing the invention can provide personalized customer service based on customer preferences and past purchase history, and can enhance the customer experience by mimicking the speaking style and tone of a particular store associate.

[0971] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0972] Step 1:

[0973] The server receives the information provided by the user and understands the profile of the person they wish to reunite with. This information includes the specific person's characteristics and past interactions. The server analyzes this information and inputs it into a generative AI model. The generative AI model generates and stores data on the profile of the person based on the input information.

[0974] Input: User-provided personal characteristics and past interactions

[0975] Data processing: Analysis and data generation using generative AI models

[0976] Output: Saved person image data

[0977] Step 2:

[0978] The server recreates realistic interactions based on the stored persona data, using a generative AI model to generate natural dialogue with the user, allowing the user to have realistic interactions with a specific person.

[0979] Input: Saved portrait data

[0980] Data Computation: Dialogue Generation with Generative AI Models

[0981] Output: The generated dialogue

[0982] Step 3:

[0983] The server uses image recognition and CG technology to visually recreate the appearance and facial expressions of a specific person, giving the user the feeling that they are visually interacting with that person.

[0984] Input: Saved portrait data

[0985] Data processing: Reproducing appearance and facial expressions using image recognition and CG technology

[0986] Output: A visually recreated human figure

[0987] Step 4:

[0988] The server generates a future image of a person that may exist, allowing the user to interact with it. The server uses a generative AI model to generate and store the future image.

[0989] Input: Information about future persona

[0990] Data Computation: Generating Future Humans with Generative AI Models

[0991] Output: Saved future persona data

[0992] Step 5:

[0993] The device allows store clerks wearing smart glasses to analyze the customer's speaking style and tone and imitate it to serve the customer. The smart glasses' microphone captures the customer's voice and sends it to a generative AI model. The generative AI model then generates a response that imitates the customer's speaking style and tone and plays it back aloud using pyttsx3.

[0994] Input: Customer voice input

[0995] Data processing: Analyzing and imitating speaking style and tone with generative AI models

[0996] Output: Audio playback of the imitated response

[0997] Step 6:

[0998] The device uses the smart glasses' camera and microphone to capture the customer's voice input in real time, then uses a generative AI model to generate responses that mimic their speaking style and tone, which are then played back aloud using pyttsx3.

[0999] Input: Customer voice input captured in real time

[1000] Data Computation: Generative AI models mimic speaking style and tone and generate responses

[1001] Output: Audio playback of the imitated response

[1002] In this way, the system can personalize customer interactions based on customer preferences and past purchases, and improve the customer experience by mimicking the speaking style and tone of a particular store associate.

[1003] Example 3

[1004] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1005] In modern society, it is physically impossible to reunite with someone from the past or future. However, advances in technology have made it possible to recreate the appearance and speaking style of a person. The present invention aims to provide a system that allows users to experience visual and auditory reunions with people from the past or future.

[1006] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for acquiring an image of a person the user wishes to reunite with, uploaded by the user; means for generating computer graphics that reproduce the person's face using a generative AI model and a prompt statement instructing the user to recreate the person's face in computer graphics based on the image of the person; means for accepting an instruction from the user that the user wishes to meet a future or past person; means for modifying the generated computer graphics using a prompt statement instructing the user to change the age of the generated computer graphics based on the accepted instruction and the generative AI model; and means for displaying the modified computer graphics. This allows the user to visually reunite with past or future people. The server also includes a means for acquiring voice data of the other person whom the user wishes to reunite with, a means for analyzing the other person's speaking style or tone from the other person's voice data, a prompt sentence instructing the user to converse with the user based on input from the user and the analysis results of the other person's speaking style or tone, a means for recreating the conversation between the user and the other person using the generative AI model, and a means for outputting the other person's remarks in the recreated conversation.

[1007] "Means for understanding the personality of the person you wish to reunite with" refers to technology that allows users to recognize the appearance and characteristics of the person they wish to reunite with and capture them as data.

[1008] "Means for reproducing realistic interactions" refers to technology that imitates the other person's speaking style, tone of voice, facial expressions, etc., to realize a dialogue with the user.

[1009] "Means of visually reproducing the other person" refers to technology that allows the appearance of the other person to be reproduced as an image or video.

[1010] "A means to meet not only past but also future counterparts" is a technology that adds or subtracts the other person's age and generates their appearance in the past or future.

[1011] "Means for users to upload photos of other users" refers to technology that allows users to use their terminals to send photos of other users to the system.

[1012] "Means for the server to receive photos and input them into the generative AI model" refers to technology that enables the server to receive photos sent by users and input them into the generative AI model in an appropriate format.

[1013] "Means for a generative AI model to recreate the other person's face using CG" refers to a technology that enables a generative AI model to recreate the other person's face using highly accurate CG based on input photographic data.

[1014] The "means for a user to indicate that he / she wishes to meet a future or past partner" is a technique for a user to input an instruction to the system that he / she wishes to meet a future or past partner.

[1015] "Means for the server to receive instructions and instruct the generative AI model to make age changes" refers to technology that enables the server to receive instructions from the user and instruct the generative AI model to perform age change calculations.

[1016] "Means by which a generative AI model generates the appearance of a partner with age changes" refers to technology that enables a generative AI model to perform age addition or subtraction calculations to generate the future or past appearance of a partner.

[1017] "Means for displaying the appearance generated by the server to the user" refers to a technique for transmitting the appearance of the other party generated by the server to the user's terminal and displaying it.

[1018] This invention is a system that allows users to experience a visual and auditory reunion with a past or future partner. In this system, the user uploads a photo of the partner, and the server receives the photo and inputs it into a generative AI model, which recreates the partner's face in CG. Furthermore, when the user indicates that they would like to meet a future or past partner, the server instructs the generative AI model to add an age change, and the generative AI model generates the partner's appearance with the age change added. Finally, the server displays the generated appearance to the user.

[1019] Specifically, the following hardware and software are used.

[1020] 1. User's device

[1021] Users access the system interface using a device such as a smartphone or PC, which has an application installed for uploading photos.

[1022] 2. Server

[1023] The server receives photos uploaded by users and inputs them into a generative AI model for image processing (e.g., StyleGAN or DeepFaceLab). The server converts the photo data into an appropriate format and passes it to the generative AI model.

[1024] 3. Generative AI Models

[1025] The generative AI model generates computer graphics that recreate the other person's face based on a prompt that instructs the model to recreate the other person's face in high-precision CG using input photo data. Specifically, it uses AI technologies such as StyleGAN and DeepFaceLab to extract facial features and generate the CG.

[1026] 4. Age Change Algorithm

[1027] The generative AI model incorporates algorithms that perform age addition or subtraction calculations, allowing it to generate the future or past appearance of a person. Based on the received instructions, the generative AI model modifies the generated computer graphics using the generative AI model in accordance with a prompt statement that instructs the model to make age changes to the generated computer graphics.

[1028] As a specific example, consider the case where the user indicates, "I want to meet the person in 10 years' time."

[1029] 1. Users upload photos of their friends

[1030] A user uses a smartphone to open the system's application, taps the "Upload Photo" button in the application, and selects and uploads a friend's photo from the gallery.

[1031] 2. The server receives the photo and inputs it into the generative AI model

[1032] The server receives the photo data and inputs it into StyleGAN, a generative AI model for image processing. The server converts the photo data into an appropriate format and passes it to the generative AI model.

[1033] 3. The generative AI model recreates the other person's face in CG

[1034] StyleGAN analyzes photo data and extracts facial features of your friend. Based on these extracted features, it generates a highly accurate CG image that realistically recreates your friend's face.

[1035] 4. The user indicates they want to meet a future or past contact.

[1036] The user enters "I want to meet my friend 10 years from now" into the input field in the application and taps the send button.

[1037] 5. The server receives the instructions and instructs the generative AI model to add age changes.

[1038] The server receives the user's instructions and instructs StyleGAN to apply an age-addition algorithm, specifically, to calculate how the friend's face will change in 10 years.

[1039] 6. The generative AI model generates the other person's appearance with age changes.

[1040] StyleGAN performs age addition calculations to generate what your friend will look like in 10 years. It realistically reproduces facial wrinkles and changes in hair color to generate your friend's future face.

[1041] 7. The server displays the generated appearance to the user

[1042] The server sends the generated image of the friend's appearance in 10 years to the user's smartphone, which displays the received data, allowing the user to see what the friend will look like in 10 years.

[1043] An example of the prompt sentence is "Please generate what your friend will look like in 10 years. I have uploaded a current photo." The flow of the identification process in the third embodiment will be described with reference to FIG.

[1044] Step 1:

[1045] Users upload photos of their contacts.

[1046] Input: The user uses the terminal to select the other person's photo file and upload it to the system interface.

[1047] Specific behavior: A user opens the application on their smartphone, taps the "Upload Photo" button, and selects and uploads a photo of a friend from their gallery.

[1048] Output: Uploaded photo data is sent to the server.

[1049] Step 2:

[1050] The server receives the photos and inputs them into a generative AI model.

[1051] Input: Photo data submitted by the user.

[1052] What it does: The server receives the photo data and converts it into a suitable format for input into a generative AI model (e.g., StyleGAN) for image processing.

[1053] Output: Photo data converted into a format that can be fed into a generative AI model.

[1054] Step 3:

[1055] The generative AI model recreates the other person's face in CG.

[1056] Input: Photo data sent from the server after format conversion.

[1057] How it works: A generative AI model (StyleGAN) analyzes photo data and extracts the facial features of the other person. Based on the extracted features, it generates highly accurate CG.

[1058] Output: CG data of the other person's face.

[1059] Step 4:

[1060] A user indicates that they would like to meet a future or past contact.

[1061] Input: The user inputs instructions into the system's interface, such as "I want to meet someone in the future" or "I want to meet someone in the past."

[1062] Specific behavior: The user enters "I want to meet my friend 10 years from now" into the input field in the application and taps the send button.

[1063] Output: User instructions are sent to the server.

[1064] Step 5:

[1065] The server receives the instructions and instructs the generative AI model to add age changes.

[1066] Input: A user-submitted instruction that they would like to meet a future or past contact.

[1067] Specific operation: The server receives the user's instructions and instructs the generative AI model to calculate the age change. Specifically, it applies an age addition algorithm to add changes to the other person's appearance in 10 years.

[1068] Output: Age change calculation instructions sent to the generative AI model.

[1069] Step 6:

[1070] The generative AI model generates the appearance of the other person, with age changes added.

[1071] Input: Age change calculation instructions sent from the server.

[1072] How it works: The generative AI model adds or subtracts age to generate the person's future or past appearance, realistically recreating changes in facial wrinkles, hair color, skin texture, and more.

[1073] Output: CG data of the opponent's appearance with age changes added.

[1074] Step 7:

[1075] The server displays the generated appearance to the user.

[1076] Input: CG data of the opponent's appearance with age changes sent from the generative AI model.

[1077] Specific operation: The server sends the generated CG data of the appearance to the user's device, which displays the received data.

[1078] Output: The future or past appearance of the other person displayed on the user's device.

[1079] The server also acquires the voice data of the person the user wishes to reunite with. The server may acquire voice data from the web based on the information of the person the user wishes to reunite with, or may acquire voice data uploaded by the user. The server then uses voice recognition technology to analyze the speaking style or tone of the other party from the other party's voice data. The server then uses a generative AI model (GPT-4) and prompts to instruct the user to converse with the other person based on the user's input and the analysis of the other person's speaking style or tone. The server then outputs the other person's utterances in the reproduced conversation to the user. This allows the user to reproduce a conversation with a future or past person they would like to meet.

[1080] The server may further include means for recognizing the user's emotions in the conversation using an emotion engine described below. In this case, the reproducing means reproduces the conversation between the user and the other party using the prompt sentence instructing the user to converse with the other party and the generative AI model based on the input from the user, the user's emotions, and the analysis result of the other party's speaking style or tone. The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for acquiring an image of a person the user wishes to reunite with, uploaded by the user; means for generating computer graphics that reproduce the person's face using a generative AI model and a prompt statement instructing the user to recreate the person's face in computer graphics based on the image of the person; means for accepting an instruction from the user that the user wishes to meet a future or past person; means for modifying the generated computer graphics using a prompt statement instructing the user to change the age of the generated computer graphics based on the accepted instruction and the generative AI model; and means for displaying the modified computer graphics. This allows the user to visually reunite with past or future people. The server also includes a means for acquiring voice data of the other person whom the user wishes to reunite with, a means for analyzing the other person's speaking style or tone from the other person's voice data, a prompt sentence instructing the user to converse with the user based on input from the user and the analysis results of the other person's speaking style or tone, a means for recreating the conversation between the user and the other person using the generative AI model, and a means for outputting the other person's remarks in the recreated conversation.

[1081] "Means for understanding the personality of the person you wish to reunite with" refers to technology that allows users to recognize the appearance and characteristics of the person they wish to reunite with and capture them as data.

[1082] "Means for reproducing realistic interactions" refers to technology that imitates the other person's speaking style, tone of voice, facial expressions, etc., to realize a dialogue with the user.

[1083] "Means of visually reproducing the other person" refers to technology that allows the appearance of the other person to be reproduced as an image or video.

[1084] "A means to meet not only past but also future counterparts" is a technology that adds or subtracts the other person's age and generates their appearance in the past or future.

[1085] "Means for users to upload photos of other users" refers to technology that allows users to use their terminals to send photos of other users to the system.

[1086] "Means for the server to receive photos and input them into the generative AI model" refers to technology that enables the server to receive photos sent by users and input them into the generative AI model in an appropriate format.

[1087] "Means for a generative AI model to recreate the other person's face using CG" refers to a technology that enables a generative AI model to recreate the other person's face using highly accurate CG based on input photographic data.

[1088] The "means for a user to indicate that he / she wishes to meet a future or past partner" is a technique for a user to input an instruction to the system that he / she wishes to meet a future or past partner.

[1089] "Means for the server to receive instructions and instruct the generative AI model to make age changes" refers to technology that enables the server to receive instructions from the user and instruct the generative AI model to perform age change calculations.

[1090] "Means by which a generative AI model generates the appearance of a partner with age changes" refers to technology that enables a generative AI model to perform age addition or subtraction calculations to generate the future or past appearance of a partner.

[1091] "Means for displaying the appearance generated by the server to the user" refers to a technique for transmitting the appearance of the other party generated by the server to the user's terminal and displaying it.

[1092] This invention is a system that allows users to experience a visual and auditory reunion with a past or future partner. In this system, the user uploads a photo of the partner, and the server receives the photo and inputs it into a generative AI model, which recreates the partner's face in CG. Furthermore, when the user indicates that they would like to meet a future or past partner, the server instructs the generative AI model to add an age change, and the generative AI model generates the partner's appearance with the age change added. Finally, the server displays the generated appearance to the user.

[1093] Specifically, the following hardware and software are used.

[1094] 1. User's device

[1095] Users access the system interface using a device such as a smartphone or PC, which has an application installed for uploading photos.

[1096] 2. Server

[1097] The server receives photos uploaded by users and inputs them into a generative AI model for image processing (e.g., StyleGAN or DeepFaceLab). The server converts the photo data into an appropriate format and passes it to the generative AI model.

[1098] 3. Generative AI Models

[1099] The generative AI model generates computer graphics that recreate the other person's face based on a prompt that instructs the model to recreate the other person's face in high-precision CG using input photo data. Specifically, it uses AI technologies such as StyleGAN and DeepFaceLab to extract facial features and generate the CG.

[1100] 4. Age Change Algorithm

[1101] The generative AI model incorporates algorithms that perform age addition or subtraction calculations, allowing it to generate the future or past appearance of a person. Based on the received instructions, the generative AI model modifies the generated computer graphics using the generative AI model in accordance with a prompt statement that instructs the model to make age changes to the generated computer graphics.

[1102] As a specific example, consider the case where the user indicates, "I want to meet the person in 10 years' time."

[1103] 1. Users upload photos of their friends

[1104] A user uses a smartphone to open the system's application, taps the "Upload Photo" button in the application, and selects and uploads a friend's photo from the gallery.

[1105] 2. The server receives the photo and inputs it into the generative AI model

[1106] The server receives the photo data and inputs it into StyleGAN, a generative AI model for image processing. The server converts the photo data into an appropriate format and passes it to the generative AI model.

[1107] 3. The generative AI model recreates the other person's face in CG

[1108] StyleGAN analyzes photo data and extracts facial features of your friend. Based on these extracted features, it generates a highly accurate CG image that realistically recreates your friend's face.

[1109] 4. The user indicates they want to meet a future or past contact.

[1110] The user enters "I want to meet my friend 10 years from now" into the input field in the application and taps the send button.

[1111] 5. The server receives the instructions and instructs the generative AI model to add age changes.

[1112] The server receives the user's instructions and instructs StyleGAN to apply an age-addition algorithm, specifically, to calculate how the friend's face will change in 10 years.

[1113] 6. The generative AI model generates the other person's appearance with age changes.

[1114] StyleGAN performs age addition calculations to generate what your friend will look like in 10 years. It realistically reproduces facial wrinkles and changes in hair color to generate your friend's future face.

[1115] 7. The server displays the generated appearance to the user

[1116] The server sends the generated image of the friend's appearance in 10 years to the user's smartphone, which displays the received data, allowing the user to see what the friend will look like in 10 years.

[1117] An example of the prompt sentence is "Please generate what your friend will look like in 10 years. I have uploaded a current photo." The flow of the identification process in the third embodiment will be described with reference to FIG.

[1118] Step 1:

[1119] Users upload photos of their contacts.

[1120] Input: The user uses the terminal to select the other person's photo file and upload it to the system interface.

[1121] Specific behavior: A user opens the application on their smartphone, taps the "Upload Photo" button, and selects and uploads a photo of a friend from their gallery.

[1122] Output: Uploaded photo data is sent to the server.

[1123] Step 2:

[1124] The server receives the photos and inputs them into a generative AI model.

[1125] Input: Photo data submitted by the user.

[1126] What it does: The server receives the photo data and converts it into a suitable format for input into a generative AI model (e.g., StyleGAN) for image processing.

[1127] Output: Photo data converted into a format that can be fed into a generative AI model.

[1128] Step 3:

[1129] The generative AI model recreates the other person's face in CG.

[1130] Input: Photo data sent from the server after format conversion.

[1131] How it works: A generative AI model (StyleGAN) analyzes photo data and extracts the facial features of the other person. Based on the extracted features, it generates highly accurate CG.

[1132] Output: CG data of the other person's face.

[1133] Step 4:

[1134] A user indicates that they would like to meet a future or past contact.

[1135] Input: The user inputs instructions into the system's interface, such as "I want to meet someone in the future" or "I want to meet someone in the past."

[1136] Specific behavior: The user enters "I want to meet my friend 10 years from now" into the input field in the application and taps the send button.

[1137] Output: User instructions are sent to the server.

[1138] Step 5:

[1139] The server receives the instructions and instructs the generative AI model to add age changes.

[1140] Input: A user-submitted instruction that they would like to meet a future or past contact.

[1141] Specific operation: The server receives the user's instructions and instructs the generative AI model to calculate the age change. Specifically, it applies an age addition algorithm to add changes to the other person's appearance in 10 years.

[1142] Output: Age change calculation instructions sent to the generative AI model.

[1143] Step 6:

[1144] The generative AI model generates the appearance of the other person, with age changes added.

[1145] Input: Age change calculation instructions sent from the server.

[1146] How it works: The generative AI model adds or subtracts age to generate the person's future or past appearance, realistically recreating changes in facial wrinkles, hair color, skin texture, and more.

[1147] Output: CG data of the opponent's appearance with age changes added.

[1148] Step 7:

[1149] The server displays the generated appearance to the user.

[1150] Input: CG data of the opponent's appearance with age changes sent from the generative AI model.

[1151] Specific operation: The server sends the generated CG data of the appearance to the user's device, which displays the received data.

[1152] Output: The future or past appearance of the other person displayed on the user's device.

[1153] The server also acquires the voice data of the person the user wishes to reunite with. The server may acquire voice data from the web based on the information of the person the user wishes to reunite with, or may acquire voice data uploaded by the user. The server then uses voice recognition technology to analyze the speaking style or tone of the other party from the other party's voice data. The server then uses a generative AI model (GPT-4) and prompts to instruct the user to converse with the other person based on the user's input and the analysis of the other person's speaking style or tone. The server then outputs the other person's utterances in the reproduced conversation to the user. This allows the user to reproduce a conversation with a future or past person they would like to meet.

[1154] The server may further include means for recognizing the user's emotions in the conversation using an emotion engine described below. In this case, the reproducing means reproduces the conversation between the user and the other party using the prompt sentence instructing the user to converse with the other party and the generative AI model based on the input from the user, the user's emotions, and the analysis result of the other party's speaking style or tone.

[1155] (Application example 3)

[1156] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1157] While conventional technology can recreate the image of a person you want to meet, it is difficult to display their appearance in the future or the past, or to try on clothes for the generated appearance. Furthermore, when users try on products in a virtual store, there is a lack of a way to check their appearance in the future or the past. This can prevent users from determining whether a product will suit their future self, potentially reducing their motivation to purchase.

[1158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1159] In this invention, the server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting the person not only in the past but also in the future, means for recreating the user's appearance and adding or subtracting age to display the future or past appearance, and means for trying on clothes with the generated appearance, thereby enabling the user to try on products in a virtual store while checking their future or past appearance.

[1160] "A means of understanding the personality of the person you wish to reunite with" is a technology that recognizes the appearance and characteristics of the person you wish to reunite with and acquires this data.

[1161] "Means for recreating realistic interactions" refers to technology that mimics actual conversations and behavior based on the acquired image of the other person.

[1162] "Means of visual reproduction" refers to technology that uses image recognition technology and CG technology to visually display the appearance of the other person.

[1163] The "means of meeting not only past but also future partners" is a technology that uses AI technology to add or subtract a partner's age and generate their appearance in the past or future.

[1164] "Means of recreating a user's appearance and adding or subtracting age to display future or past appearances" refers to technology that uses AI technology to change the age based on the user's own appearance and display future or past appearances.

[1165] The "means for trying on clothes for the generated appearance" is a technology for virtually trying on selected clothes for the generated appearance.

[1166] The following system configuration is used as an embodiment of the present invention.

[1167] System Configuration

[1168] The system consists of a server, a user terminal, and various software components. The server understands the person's profile, recreates realistic interactions, and processes various data to visually recreate them. The user terminal is a device such as a smartphone or head-mounted display, and provides an interface for users to interact with the system.

[1169] Hardware and software used

[1170] Hardware: Smartphones, head-mounted displays, servers

[1171] Software: Python, OpenCV, DeepFace, PIL, ageitgey / face_recognition library

[1172] Data processing and calculation

[1173] 1. How to understand the person you want to reunite with:

[1174] The server receives the photo of the person uploaded by the user and uses image recognition technology to extract the person's appearance and features, using OpenCV and the ageitgey / face_recognition library.

[1175] 2. Means of replicating real-life interactions:

[1176] The server uses voice recognition technology to mimic the other person's speaking style and tone based on the extracted characteristics of the other person, giving the user an experience that feels like they are actually having a conversation with the other person.

[1177] 3. Visual Representation:

[1178] The server uses image recognition and CG technology to visually recreate the other person's appearance, using the DeepFace library to recreate the other person's face with high accuracy.

[1179] 4. Ways to meet not only past but future partners:

[1180] The server uses AI technology to add or subtract the partner's age to generate past and future appearances, allowing users to see what their partner will look like in the future by entering a prompt such as "I want to meet my partner in 10 years."

[1181] 5. A way to recreate the user's appearance and add or subtract age to show future or past appearances:

[1182] The server uses AI technology to age the user based on their own photo, generating future and past appearances, allowing users to see what they will look like in the future.

[1183] 6. Means of trying on clothes for generated looks:

[1184] The server virtually tries on the selected clothing for the generated look, and uses the PIL library to overlay images of the clothing onto the user's look.

[1185] Specific examples

[1186] If the user says, "I want to see how this outfit will look on me in 10 years," enter the following prompt text:

[1187] "I want to see if this outfit will look good on me in 10 years."

[1188] The server receives this prompt and generates a visual of the user's appearance in 10 years' time based on the user's current photo, and then displays the selected clothing on the visual, allowing the user to check the suitability of the product for their future self.

[1189] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1190] Step 1:

[1191] The user uses the device to upload a photo of the person they want to meet again.

[1192] Input: Photo of the person

[1193] Output: The photo data of the other person sent to the server

[1194] Specific operation: The user uses a smartphone or head-mounted display to take or select a photo of the person they want to reunite with, and uploads it to the server through the application.

[1195] Step 2:

[1196] The server analyzes the received photos and extracts the other person's appearance and characteristics.

[1197] Input: Uploaded photo data of the other person

[1198] Output: Extracted appearance and feature data of the other person

[1199] Specific operation: The server uses OpenCV and the ageitgey / face_recognition library to extract facial features from the photo and obtain data on the other person's appearance and characteristics.

[1200] Step 3:

[1201] The server uses the extracted features to recreate the other person's appearance using CG.

[1202] Input: Extracted appearance and feature data of the other person

[1203] Output: Reconstructed CG image of the other person

[1204] Specific operation: The server uses the DeepFace library to accurately recreate the other person's face based on the extracted features and generates it as a CG image.

[1205] Step 4:

[1206] A user uses a terminal to input a prompt sentence such as "I want to meet my partner in 10 years' time."

[1207] Input: Prompt sentence (e.g., "I want to meet the person in 10 years' time")

[1208] Output: Prompt text sent to the server

[1209] Specific operation: The user inputs a prompt message through the application interface indicating that they would like to meet a prospective partner and sends it to the server.

[1210] Step 5:

[1211] The server parses the prompt and adds or subtracts the person's age to generate a future or past appearance.

[1212] Input: Prompt text, CG image of the other person

[1213] Output: CG image of the other person from the future or past

[1214] What it does: The server uses a generative AI model to add or subtract the person's age based on the prompt text, generating future and past appearances.

[1215] Step 6:

[1216] The user uses the terminal to select an image of the clothing item that they would like to try on.

[1217] Input: Clothing image

[1218] Output: Image data of the garment sent to the server

[1219] Specific operation: Through the application interface, the user selects an image of the clothing they want to try on and uploads it to the server.

[1220] Step 7:

[1221] The server then tries on the selected clothing for the generated look.

[1222] Input: CG images of future or past opponents, clothing image data

[1223] Output: CG image of the person trying on the clothing

[1224] Specific operation: The server uses the PIL library to overlay an image of the clothing on the generated appearance and generates a CG image of the clothing being tried on.

[1225] Step 8:

[1226] The server sends a CG image of the garment being tried on to the user's terminal and displays it.

[1227] Input: CG image of someone trying on clothing

[1228] Output: CG image of the garment being tried on, displayed on the user's device

[1229] Specific operation: The server sends the generated CG image of the garment being tried on to the user's device, and the user can view the image via a smartphone or head-mounted display.

[1230] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1231] "Example 1"

[1232] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes the user's emotions and adjusts interactions with a person's image of a person the user wishes to reunite with based on those emotions. Specifically, the system recognizes emotions from the user's voice and facial expressions and adjusts the reaction of the other person's image based on those emotions. For example, if the user expresses joy, the system causes the other person's image to generate a joyful reaction. Conversely, if the user expresses sadness, the system causes the other person's image to generate a comforting reaction.

[1233] "Example 2"

[1234] In another embodiment of the present invention, a system is provided in which an emotion engine adjusts the future image of the other person in response to the user's emotions. Specifically, if the user expresses emotions of expectation or hope, the system adjusts the future image of the other person to be brighter and more positive. For example, if the user indicates, "I want to meet my partner in 10 years," and expresses emotions of expectation, the system displays the other person's future image in 10 years as successful and happy.

[1235] "Example 3"

[1236] Furthermore, as another embodiment of the present invention, a system is provided in which an emotion engine adjusts the past appearance of a partner according to the user's emotion. Specifically, if the user expresses feelings of nostalgia or nostalgia, the system adjusts the past appearance of the partner to be more memorable and moving. For example, if the user indicates, "I want to meet the partner five years ago," and expresses nostalgia, the system displays the partner's appearance five years ago as a special memorable scene.

[1237] The processing flow of each embodiment will be described below.

[1238] "Example 1"

[1239] Step 1: Receive input from the user (voice, facial expressions, etc.).

[1240] Step 2: The emotion engine recognizes the user's emotion.

[1241] Step 3: Tailor your interactions with the persona you want to reunite with based on the emotions you recognize.

[1242] "Example 2"

[1243] Step 1: Receive input from the user (voice, facial expressions, etc.).

[1244] Step 2: The emotion engine recognizes the user's emotion.

[1245] Step 3: If the user expresses feelings of expectation or hope, the system adjusts the other person's view of the future to be brighter and more positive.

[1246] "Example 3"

[1247] Step 1: Receive input from the user (voice, facial expressions, etc.).

[1248] Step 2: The emotion engine recognizes the user's emotion.

[1249] Step 3: If the user expresses feelings of nostalgia or nostalgia, the system adjusts the person's past appearance to make it more memorable and moving.

[1250] Example 1

[1251] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1252] With conventional technology, it was difficult to realistically recreate the image of the person you wanted to reunite with, and it was not possible to adjust interactions based on the user's emotions. In addition, there was a lack of means to meet not only past but also future contacts. This made it difficult for users to experience more realistic interactions with people they wanted to reunite with.

[1253] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1254] In this invention, the server includes means for collecting information provided by the user, means for generating a person's portrait based on the information using artificial intelligence, and means for recognizing the user's emotions and adjusting interactions with the person's portrait in accordance with the emotions. This makes it possible to realistically recreate the person the user wants to reunite with and adjust interactions in accordance with the user's emotions.

[1255] "Means for collecting information provided by users" refers to devices or software for collecting photos, videos, audio, text information, etc. of the person with whom the user wishes to reunite.

[1256] "Means for generating a person's profile using artificial intelligence" refers to algorithms or programs that learn about the other person's appearance, speaking style, personality, preferences, etc. based on collected information, and generate a profile of that person.

[1257] "Means for recognizing the user's emotions and adjusting interactions with the other person's character in accordance with those emotions" refers to a system that analyzes the user's voice and facial expressions, recognizes the user's emotions, and adjusts the other person's character's response based on those emotions.

[1258] "Means of visual reproduction" refers to devices or software that use image recognition technology and computer graphics technology to imitate and visually reproduce the appearance of another person.

[1259] "A means to meet not only past but also future partners" is a system that generates images of not only past but also future partners, allowing users to interact with those images.

[1260] "Voice recognition technology" is a technology that analyzes the voice of the user or other person and imitates their speaking style and tone.

[1261] "Image recognition technology" is a technology that analyzes the appearance of others from photographs and videos and extracts their characteristics.

[1262] "Computer graphics technology" is a technology for visually reproducing the image of a person based on analyzed external features.

[1263] The present invention provides a system that realistically recreates the image of a person with whom a user wants to reunite and adjusts interactions according to the user's emotions. Specific embodiments of this system will be described below.

[1264] First, the user provides photos, videos, audio, and text information about the person they want to reunite with. This information is collected through devices such as smartphones and PCs. The devices then send the photos taken by the user and the audio recorded by the user to a server via an application.

[1265] The server receives the information sent from the device and stores it in a database. For example, photo data is stored in an image folder, and audio data is stored in an audio folder. The server then inputs the stored information into an artificial intelligence (AI) model. This AI model may be, for example, OpenAI's GPT-4 or DeepMind's DALL-E.

[1266] The AI ​​model generates an image of the other person based on the input information. Specifically, it extracts the other person's facial features from photos and learns their speaking style from their audio. This allows it to reproduce the other person's appearance, speaking style, personality, preferences, etc.

[1267] Furthermore, an emotion engine is built in to recognize the user's emotions. The device collects the user's voice and facial expressions through a camera and microphone and sends them to the emotion engine. The emotion engine analyzes the collected voice and facial expression data to recognize the user's emotions. For example, it can determine whether the user is happy or sad from changes in voice tone and facial expression.

[1268] The server adjusts the reaction of the other person's figure based on the emotion information received from the emotion engine. For example, if the user expresses a happy emotion, the server makes the other person's figure generate a happy reaction. Conversely, if the user expresses a sad emotion, the server makes the other person's figure generate a comforting reaction.

[1269] Finally, the device displays the reaction of the other person's image received from the server to the user. For example, the other person's image may be displayed with a smiling face on the smartphone screen. In this way, the system adjusts the interaction with the other person's image according to the user's emotions.

[1270] As a concrete example, consider the case where a user provides a photo and voice of the person they want to reunite with. The user uses their smartphone to upload the photo and voice of the person to the system. The server receives this data and inputs it into an AI model. The AI ​​model learns the person's appearance and speaking style and generates an image of that person.

[1271] Next, the user starts a conversation with the system. When the user says, "I'm feeling very happy today," the device sends this speech to the emotion engine. The emotion engine recognizes the user's happy emotion and sends that information to the server. The server then generates a happy reaction in the other person's character and displays it to the user.

[1272] Example prompt sentence:

[1273] Users provide a photo and voice of the person they want to reunite with, and the system generates a portrait of that person. When the user says, "I'm feeling very happy today," the system recognizes the user's emotion and generates a happy reaction in the portrait of the person they want to reunite with.

[1274] In this way, the system can tailor its interactions with the other persona depending on the user's emotions.

[1275] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1276] Step 1:

[1277] User provides information

[1278] Users provide photos, videos, audio, and text information about the person they want to reunite with. For example, they can take a photo of the person with their smartphone camera and record a voice message with the microphone. This information is used as input.

[1279] Step 2:

[1280] The device collects information

[1281] The device collects photos and audio provided by the user through the application. Specifically, it sends the photo data taken by the user and the audio data recorded by the user to the server. The input is the information provided by the user, and the output is the data sent to the server.

[1282] Step 3:

[1283] The server stores the information

[1284] The server receives the information sent from the device and stores it in a database. For example, photo data is stored in an image folder and audio data is stored in an audio folder. The input is the data sent from the device, and the output is the information stored in the database.

[1285] Step 4:

[1286] The server inputs information into the AI ​​model

[1287] The server inputs the stored information into an AI model. Specifically, photo data and audio data are input into an AI model (such as GPT-4 or DALL-E) to learn the other person's appearance and speaking style. The input is the information stored in the database, and the output is the learning result of the AI ​​model.

[1288] Step 5:

[1289] AI model generates a portrait of the other person

[1290] The AI ​​model generates a portrait of the other person based on the input information. Specifically, it extracts the other person's facial features from a photo and learns speaking characteristics from audio. The input is data provided by the server, and the output is the generated portrait of the other person.

[1291] Step 6:

[1292] The device collects the user's emotions

[1293] The device collects the user's voice and facial expressions through a camera and microphone. For example, the microphone records the user's speech and the camera captures their facial expressions. The input is the user's voice and facial expressions, and the output is the collected emotional data.

[1294] Step 7:

[1295] Emotion engine analyzes emotions

[1296] The emotion engine analyzes collected voice and facial expression data to recognize the user's emotions. Specifically, it determines whether the user is happy or sad based on changes in voice tone and facial expression. The input is the collected emotional data, and the output is the analyzed emotional information.

[1297] Step 8:

[1298] The server generates a response based on the emotion.

[1299] The server adjusts the reaction of the other person's figure based on the emotional information received from the emotion engine. For example, if the user expresses joy, the server makes the other person's figure generate a joyful reaction. The input is the analyzed emotional information, and the output is the adjusted reaction of the other person.

[1300] Step 9:

[1301] The device displays the response to the user

[1302] The device displays the reaction of the other person's image received from the server to the user. For example, the other person's image is displayed with a smiling face on the screen of a smartphone. The input is the reaction data sent from the server, and the output is the other person's image displayed to the user.

[1303] (Application example 1)

[1304] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1305] Conventional virtual shopping experiences have struggled to provide personalized support to users, particularly in terms of responding to the user's emotions. Furthermore, there is a lack of technology to realistically recreate the person the user wants to meet and adjust their response based on the user's emotions.

[1306] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for recognizing the user's emotions and adjusting the reaction of the personality of the person in accordance with the emotions, and means for functioning as a shopping assistant in a virtual store. This makes it possible to provide the user with a personalized shopping experience and respond according to the user's emotions.

[1307] "Profile of the person you wish to meet again" refers to information including characteristics such as appearance, speaking style, personality, and preferences of a specific person that the user wishes to meet again.

[1308] "Means of understanding" refers to technology that uses artificial intelligence to analyze and learn about the other person's personality based on information provided by the user (photos, videos, audio, text information, etc.).

[1309] "Means of recreating realistic interactions" refers to technology that uses a generated image of the other person to mimic actual conversations and communication with the user.

[1310] "Means of visual reproduction" refers to technology that uses image recognition technology and computer graphics technology to realistically reproduce the appearance of the other person.

[1311] "A means to meet future partners" refers to technology that not only uses past information, but also generates and reproduces the image of a person you would like to meet again in the future.

[1312] "Means of recognizing emotions" refers to technology that analyzes emotions from the user's voice and facial expressions and understands those emotions.

[1313] "Means for adjusting the reaction of the other person's character according to their emotions" refers to a technology that changes the reaction of the generated other person's character based on the user's recognized emotions.

[1314] "Virtual store" refers to a virtual store operated on the Internet, which provides an environment where users can browse and purchase products online.

[1315] "Means to function as a shopping assistant" refers to technology in which a generated persona provides information and advice about products to support the user's shopping experience.

[1316] The system for implementing this invention has the function of understanding the profile of the person the user wants to reunite with and adjusting the response of that profile according to the user's emotions. A specific embodiment of this system will be described below.

[1317] System Configuration

[1318] The system consists of the following major components:

[1319] 1. User terminal: A device such as a smartphone or head-mounted display.

[1320] 2. Server: A central server that hosts AI models and databases.

[1321] 3. Camera: A camera to capture the user's facial expressions and movements.

[1322] 4. Microphone: A microphone to capture the user's voice.

[1323] Hardware and software used

[1324] Hardware: smartphone, head-mounted display, camera, microphone.

[1325] Software: OpenCV (acquisition and display of camera images), EmotionRecognizer (library for emotion recognition), AI PersonaGenerator (library for generating person images using AI).

[1326] Data processing and calculation

[1327] 1. User Device:

[1328] Collect information provided by users (photos, videos, audio, text information).

[1329] It uses a camera and microphone to capture the user's facial expressions and voice in real time.

[1330] 2. Server:

[1331] Based on the collected user information, an AI model is used to generate a profile of the person the user would like to meet again.

[1332] EmotionRecognizer is used to recognize emotions from captured user facial expressions and voice.

[1333] Based on the recognized emotions, the AI ​​PersonaGenerator is used to adjust the reactions of the other person's character.

[1334] Specific examples

[1335] While users are shopping in a virtual store, the image of the person they want to meet will happily say, "This product is perfect for you!" If the user is tired, the image will gently say, "Shall we take a break?"

[1336] Prompt Sentence Examples

[1337] Generate a persona of the person you want to reunite with based on photos, videos, audio, and text information provided by the user. The generated persona should adjust its reaction according to the user's emotions. If the user is happy, show a happy reaction, and if the user is sad, show a comforting reaction.

[1338] This system allows users to enjoy a more personalized shopping experience in a virtual store, and responds to users' emotions, making interactions more natural and friendly.

[1339] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1340] Step 1:

[1341] The user device collects information provided by the user (photos, videos, audio, and text information). This information becomes input data for generating a profile of the person the user wants to meet. Specifically, the user uses a smartphone or head-mounted display to take photos, record videos and audio, and input text information.

[1342] Step 2:

[1343] The user terminal sends the collected information to the server, which receives the information and stores it in a database for subsequent processing.

[1344] Step 3:

[1345] Based on the received information, the server uses a generative AI model to generate a portrait of the person the user wants to meet. Specifically, it uses the AI ​​PersonaGenerator library to analyze and integrate appearance from photos and videos, speaking style from audio, and personality and preferences from text information. The input for this process is the information provided by the user, and the output is a generated portrait.

[1346] Step 4:

[1347] The user device uses a camera and microphone to capture the user's facial expressions and voice in real time, and the captured data becomes input data for recognizing the user's emotions.

[1348] Step 5:

[1349] The user device sends the captured facial and voice data to the server, which receives the data and analyzes the user's emotions using the EmotionRecognizer library. The input of this process is the captured facial and voice data, and the output is the recognized emotion.

[1350] Step 6:

[1351] The server adjusts the reaction of the generated character based on the recognized emotion. Specifically, it uses the AI ​​PersonaGenerator library to generate a reaction according to the user's emotion. For example, if the user is happy, the character will show a happy reaction, and if the user is sad, the character will show a comforting reaction. The input of this process is the recognized emotion, and the output is the adjusted reaction.

[1352] Step 7:

[1353] The user device displays the adjusted response to the user. Specifically, the generated character is displayed on the screen of a smartphone or head-mounted display, speaking to the user. The input of this process is the adjusted response, and the output is the video and audio displayed to the user.

[1354] These steps allow users to enjoy a more personalized shopping experience in virtual stores, and also enable responses based on users' emotions, making interactions more natural and friendly.

[1355] Example 2

[1356] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1357] Conventional systems have difficulty realistically recreating the image of the person you want to reunite with, especially predicting and displaying their future appearance. Furthermore, they lacked the functionality to adjust the other person's future appearance based on the user's emotions, limiting the user experience. Furthermore, the technology required for realistic dialogue was insufficient, making it difficult to achieve natural interactions with users.

[1358] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1359] In this invention, the server includes means for receiving user input, means for transmitting input data to the server, means for generating a response using a generative AI model, means for analyzing the user's emotions using an emotion engine and adjusting the other person's future appearance, means for transmitting the response data to the user terminal, and means for displaying the response to the user. This makes it possible to adjust the future appearance according to the user's emotions and to reproduce realistic dialogue.

[1360] A "means for receiving user input" is an interface through which a user provides information to the system in the form of text, voice, or other information.

[1361] "Means for sending input data to the server" refers to a communication means for transferring information provided by the user to the server.

[1362] A "means for generating responses using a generative AI model" is a means for using a pre-trained artificial intelligence model to generate appropriate responses based on user input.

[1363] "Means of using an emotion engine to analyze the user's emotions and adjust the other person's future appearance" is a technology that analyzes the user's emotional state and appropriately changes the other person's future appearance based on the results.

[1364] The "means for transmitting response data to a user terminal" refers to a communication means for transferring the generated response to the user's device.

[1365] A "means for displaying a response to a user" is an interface for visually or audibly displaying the generated response on the user's device.

[1366] "Means of recreating realistic interactions" refers to technology that mimics the speaking style and tone of the other person to achieve natural dialogue with the user.

[1367] "Visual reproduction means" refers to a technique that uses image recognition technology and computer graphics technology to realistically reproduce the appearance of the other person.

[1368] "A means to meet not only past but also future partners" is a technology that allows users to visually see not only past partners but also future partners.

[1369] The present invention provides a system that realistically recreates the image of a person with whom a user wishes to reunite, allowing the user to meet not only past but also future contacts. A specific embodiment of this system will be described below.

[1370] Hardware and software used

[1371] Hardware: High-performance servers, user devices (PCs, smartphones)

[1372] Software: Generative AI models (e.g., GPT-4), emotion engines, dialogue management systems

[1373] System Overview

[1374] The system receives user input, sends it to a server, and generates a response using a generative AI model. It then analyzes the user's emotions using an emotion engine and adjusts the other person's future appearance. The generated response data is sent to the user's device and displayed to the user.

[1375] Specific examples

[1376] 1. User Input

[1377] A user opens the app on their smartphone, types "Good morning" into the text box, and presses the send button.

[1378] 2. Sending input data

[1379] The terminal encrypts the text data "Good morning" and sends it to a server via the Internet.

[1380] 3. Response generation using generative AI models

[1381] The server passes the received data "Good morning" to the generative AI model, and the model generates a text response saying "Good morning."

[1382] 4. Emotion analysis using an emotion engine

[1383] The server then passes the generated "Good morning" response to the emotion engine, which analyzes the user's emotions. For example, if a user inputs "I want to meet my partner in 10 years' time," expressing a feeling of anticipation, the emotion engine will analyze that emotion and adjust the partner's future appearance.

[1384] 5. Sending response data

[1385] The server encrypts the generated response data "Good morning" and sends it to the user's terminal.

[1386] 6. Displaying a response to the user

[1387] The device decodes the received response data, "Good morning," and displays it on the screen. If the user instructs, "I want to meet my partner in 10 years," the device will display the partner in 10 years' time as a successful and happy person.

[1388] Prompt Sentence Examples

[1389] "If a user says 'Good morning,' please reply 'Good morning' in the other person's voice."

[1390] "If a user indicates, 'I want to see my partner in 10 years,' and expresses an emotion of anticipation, show them as successful and happy in 10 years."

[1391] In this way, the server, terminal, and user each play their respective roles, creating a system that enables real dialogue and coordination of the future.

[1392] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1393] Step 1:

[1394] The user uses the device to enter text.

[1395] Specifically, the user opens the app on their smartphone, types "Good morning" into the text box, and presses the send button.

[1396] Input: User text input "Good morning"

[1397] Output: Text data entered into the terminal

[1398] Step 2:

[1399] The terminal transmits the user's input data to the server.

[1400] Specifically, the terminal encrypts the text data "Good morning" and sends it to a server via the Internet.

[1401] Input: Text data entered into the terminal: "Good morning"

[1402] Output: Encrypted text data sent to the server

[1403] Step 3:

[1404] The server passes the received input data to the generative AI model and generates a response.

[1405] Specifically, the server passes the data "Good morning" to a generative AI model (e.g., GPT-4), and the model generates a text response saying "Good morning."

[1406] Input: Encrypted text data "Good morning" sent to the server

[1407] Output: The response text "Good morning" generated by the generative AI model

[1408] Step 4:

[1409] The server passes the generated response to the emotion engine to analyze the user's emotion.

[1410] Specifically, the server passes the generated response "Good morning" to the emotion engine, which then analyzes the user's emotions. For example, if a user inputs "I want to meet my partner in 10 years' time," expressing a feeling of anticipation, the emotion engine will analyze that emotion and adjust the partner's future appearance.

[1411] Input: The response text "Good morning" generated by the generative AI model

[1412] Output: User emotion data analyzed by the emotion engine

[1413] Step 5:

[1414] The server transmits the generated response data to the user terminal.

[1415] Specifically, the server encrypts the generated response data "Good morning" and sends it to the user's terminal.

[1416] Input: User emotion data analyzed by the emotion engine and the response text "Good morning" generated by the generative AI model

[1417] Output: Encrypted response data sent to the user's device

[1418] Step 6:

[1419] The terminal displays the received response data to the user.

[1420] Specifically, the device decodes the received response data, "Good morning," and displays it on the screen. If the user instructs, "I want to meet my partner in 10 years," the device will display the partner in 10 years' time as a successful and happy person.

[1421] Input: Encrypted response data sent to the user terminal

[1422] Output: The response text "Good morning" displayed to the user and the person's future appearance

[1423] (Application example 2)

[1424] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1425] Conventional virtual stores have faced the challenge of monotonous dialogue with users and a lack of flexibility to respond to their emotions and expectations. Furthermore, they lacked the means to help users imagine future usage scenarios, making it difficult to stimulate purchasing motivation.

[1426] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1427] In this invention, the server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting the person not only in the past but also in the future, means for recognizing the user's emotions, means for adjusting the person's future appearance according to the user's emotions, means for introducing products in a virtual store, means for generating responses based on the user's input, and means for displaying the generated responses. This enables flexible responses according to the user's emotions and expectations, and by specifically depicting future usage scenarios, it is possible to increase the user's desire to purchase.

[1428] "Means to understand the personality of the person you wish to reunite with" refers to technology that allows users to collect and analyze the characteristics and information of the person they wish to reunite with.

[1429] "Means of recreating real interactions" refers to technology that allows natural communication, like an actual conversation, based on collected personal profiles.

[1430] "Visual reproduction means" refers to techniques that use image recognition technology and computer graphics technology to realistically reproduce the appearance of a target person.

[1431] "A means to meet not only past but also future counterparts" is a technology that allows users to visually see not only past but also future images of people.

[1432] "Means for recognizing user emotions" refers to technology for analyzing and recognizing emotions from a user's facial expressions and voice.

[1433] "Means for adjusting the future appearance of the other person according to the user's emotions" is a technology for dynamically changing the future appearance of the other person based on the user's emotional state.

[1434] "Means for introducing products in a virtual store" refers to technology for providing and explaining product information to users in a virtual environment.

[1435] "Means for generating a response based on user input" refers to technology for generating an appropriate response to a question or request from a user.

[1436] A "means for displaying the generated response" is a technique for visually or audibly presenting the generated response to the user.

[1437] A system for carrying out this invention has the following configuration: A server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting the person not only in the past but also in the future, means for recognizing the user's emotions, means for adjusting the person's future appearance according to the user's emotions, means for introducing products in a virtual store, means for generating a response based on a user's input, and means for displaying the generated response.

[1438] Program processing explanation

[1439] The server first collects and analyzes information provided by the user to understand the person the user wants to reunite with. This information includes the other person's appearance, voice, speaking style, and past events. Next, based on the collected information, it uses voice recognition technology and computer graphics technology to recreate a realistic interaction. This makes it possible to mimic the other person's speaking style and tone and visually reproduce them.

[1440] Furthermore, to recognize the user's emotions, the system uses a camera and microphone to analyze the user's facial expressions and voice. Using emotion recognition technology, the system grasps the user's emotional state in real time. Based on this emotional information, the system adjusts the future image of the other person. For example, if the user expresses feelings of expectation, the system will display the other person's future image as successful and happy.

[1441] In the virtual store, when a user asks a question about a product, the server uses a generative AI model to generate an appropriate response. Based on the user's input, the server generates a prompt and uses the OpenAI API to generate a response. The generated response is presented to the user visually or audibly.

[1442] Specific examples

[1443] For example, if a user asks, "Tell me about this product," the server will explain, "This product uses the latest technology and is very high-performance." If a user expresses anticipation by asking, "What will happen if I wear this outfit?", the server will describe a future scene, saying, "When you wear this outfit, you will look very attractive and will be praised by those around you."

[1444] Prompt Sentence Examples

[1445] A user asks, "Tell me about this product." The user emotion is "Interested."

[1446] In this way, it becomes possible to respond flexibly to the user's emotions and expectations, and by specifically describing future usage scenarios, it is possible to increase the user's desire to purchase.

[1447] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1448] Step 1:

[1449] The server collects and analyzes information provided by users to understand the person they wish to reunite with. As input, it receives information provided by users about the other person's appearance, voice, speaking style, and past events. This information is stored in a database and an analysis algorithm is used to extract the other person's characteristics. As output, it obtains the other person's characteristic data.

[1450] Step 2:

[1451] The server uses voice recognition and computer graphics technologies to recreate realistic interactions based on the collected feature data of the other party. As input, it uses the feature data of the other party obtained in step 1. It uses voice recognition technology to imitate the other party's speaking style and tone, and computer graphics technology to recreate the other party's appearance. As output, it generates a model of the other party with which it can have realistic interactions.

[1452] Step 3:

[1453] The server uses a camera and microphone to analyze the user's facial expressions and voice to recognize their emotions. It receives camera footage and audio data as input. It uses emotion recognition technology to analyze the user's emotional state in real time. The output is the user's emotional data.

[1454] Step 4:

[1455] The server adjusts the other person's future appearance based on the user's emotional data. As input, it uses the user's emotional data obtained in step 3 and the other person's model generated in step 2. If the user shows an expectant emotion, it adjusts the other person's future appearance as successful and happy. As output, it obtains the adjusted future model of the other person.

[1456] Step 5:

[1457] When a user asks a question about a product in the virtual store, the server uses a generative AI model to generate an appropriate response. As input, it receives the user's question and the user's emotion data obtained in step 3. It generates a prompt sentence and generates a response using the OpenAI API. As output, it obtains the generated response.

[1458] Step 6:

[1459] The server presents the generated response to the user visually or audibly. It uses the generated response obtained in step 5 as input. It presents the response to the user using a display or speaker. As output, the user can confirm the response.

[1460] In this way, it becomes possible to respond flexibly to the user's emotions and expectations, and by specifically describing future usage scenarios, it is possible to increase the user's desire to purchase.

[1461] Example 3

[1462] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1463] With current technology, it is difficult to realistically recreate the appearance and speaking style of a person in response to a user's request to meet a past or future partner. It is also difficult to adjust the other person's appearance in response to the user's emotions. Furthermore, the process of generating the other person's appearance data using a generative AI model and recreating it using CG technology is complex and requires efficient execution.

[1464] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting not only past but also future people, means for adjusting the appearance of the person in response to the user's emotions, means for generating appearance data of the person using a generative AI model, means for generating a prompt message and sending it to the generative AI model, and means for sending the generated appearance data to CG generation software to recreate the appearance of the person. This allows the user to have the experience of meeting a person from the past or future.

[1465] "Means for understanding the personality of the person you wish to reunite with" refers to technology that allows users to analyze the characteristics and information of the person they wish to reunite with and obtain this data.

[1466] "Means for reproducing realistic interactions" refers to technology that uses acquired information about the other party to imitate the other party's speaking style and tone, and realistically reproduces a conversation with the user.

[1467] "Means of visual reproduction" refers to technology that uses image recognition technology and CG technology to reproduce the appearance of the other person and present it visually to the user.

[1468] "A means to meet not only past but also future partners" is a technology that adds or subtracts the other person's age and generates the appearance of the past or future partner.

[1469] The "means for adjusting the appearance of the other party according to the user's emotions" is a technology for analyzing the user's emotions and adjusting the appearance of the other party according to those emotions.

[1470] "Means for generating appearance data of another party using a generative AI model" refers to technology for generating data about the appearance of another party using a generative AI model.

[1471] "Means for generating a prompt sentence and sending it to a generative AI model" refers to technology for generating a prompt sentence based on a user's instructions and sending the prompt sentence to a generative AI model.

[1472] "Means of sending the generated appearance data to CG generation software and reproducing the other person's appearance" refers to a technology for sending appearance data obtained from a generative AI model to CG generation software and reproducing the other person's appearance based on that data.

[1473] This invention is a system that realistically recreates the appearance and speaking style of a person in response to a user's desire to meet a past or future partner. This system includes multiple means for understanding the personality of the person the user wishes to reunite with, recreating realistic interactions, and visually recreating the person.

[1474] Hardware and software used

[1475] Hardware: High-performance servers, user devices (PCs, smartphones, etc.)

[1476] Software: Generative AI models (e.g., GPT-4), CG generation software (e.g., Blender), emotion engines (e.g., Affectiva)

[1477] Specific operation of the system

[1478] 1. Receive user instructions

[1479] A user uses a terminal to input an instruction to meet a person in the past or future. For example, the user inputs "I want to meet the person in 10 years' time."

[1480] 2. The server receives the instruction

[1481] The server receives instructions from the user and analyzes their content. For example, it analyzes an instruction such as "I want to meet my partner in 10 years' time."

[1482] 3. The server generates a prompt

[1483] The server generates a prompt to send to the generative AI model based on the instructions. For example, it creates a prompt such as "Please generate what the other person will look like in 10 years' time."

[1484] 4. The server sends a prompt to the generative AI model

[1485] The server sends the generated prompt to the generative AI model. For example, it sends the prompt to GPT-4: "Generate what the other person will look like in 10 years."

[1486] 5. The generative AI model generates the other person's appearance data

[1487] The generative AI model generates appearance data based on the prompt, for example, what the person's facial features will be like in 10 years.

[1488] 6. The server sends the appearance data to the CG generation software

[1489] The server sends the appearance data received from the generative AI model to CG generation software, for example, Blender, to see what the other person will look like in 10 years.

[1490] 7. Generative computer graphics software recreates the appearance of your opponent.

[1491] CG generation software then uses the received data to recreate the other person's appearance. For example, Blender creates a 3D model of what the other person will look like in 10 years' time.

[1492] 8. The server uses the emotion engine to adjust the appearance (if necessary)

[1493] If the user shows a certain emotion, the server uses the emotion engine to adjust the appearance, for example, if the user shows nostalgia, the emotion engine analyzes the emotion and adjusts the appearance as a special memorable scene.

[1494] 9. The server displays the final look to the user

[1495] The server then displays the final generated appearance on the user's device, for example, displaying the partner's face 10 years from now on the user's smartphone.

[1496] Examples and prompts

[1497] Example 1: When the user says "I want to meet my partner in 10 years"

[1498] Prompt: "Generate what your partner will look like in 10 years."

[1499] Processing: The server sends the prompt to the generative AI model, and the CG generation software uses the generated data to recreate the other person's appearance 10 years in the future.

[1500] Example 2: The user indicates nostalgia by saying, "I want to meet someone from five years ago."

[1501] Prompt: "Generate what your partner looked like five years ago and reflect your feelings of nostalgia."

[1502] Processing: The server sends the prompt text to the generative AI model, and based on the generated data, the CG generation software recreates the appearance of the other person five years ago, and the emotion engine adjusts a special memorable scene that reflects the emotion of nostalgia.

[1503] In this way, the user can have the experience of meeting a partner from the past or future. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1504] Step 1:

[1505] The user enters instructions

[1506] The user uses the terminal to input instructions about wanting to meet a partner in the past or future. For example, they can input "I want to meet the partner in 10 years' time" by voice. The input instructions are sent to the server as text data.

[1507] Step 2:

[1508] The server receives the instruction

[1509] The server receives instructions from the user and analyzes their content. For example, it analyzes the instruction "I want to meet the person I'll be with in 10 years" and understands the instruction content. The input is the user's instruction text, and the output is the analyzed instruction content.

[1510] Step 3:

[1511] The server generates a prompt

[1512] The server generates a prompt to send to the generative AI model based on the instructions. For example, it creates a prompt such as "Please generate what the other person will look like in 10 years." The input is the parsed instructions, and the output is the generated prompt.

[1513] Step 4:

[1514] The server sends a prompt to the generative AI model

[1515] The server sends the generated prompt to the generative AI model. For example, it sends the prompt to the generative AI model, "Please generate what the other person will look like in 10 years." The input is the generated prompt, and the output is the result sent to the generative AI model.

[1516] Step 5:

[1517] A generative AI model generates the other person's appearance data

[1518] The generative AI model generates appearance data based on the prompt. For example, it generates the facial features of the other person 10 years from now. The input is the prompt, and the output is the generated appearance data.

[1519] Step 6:

[1520] The server sends the appearance data to the CG generation software.

[1521] The server sends the appearance data received from the generative AI model to the CG generation software. For example, it sends data on the other person's face 10 years from now to the CG generation software. The input is the generated appearance data, and the output is the result sent to the CG generation software.

[1522] Step 7:

[1523] CG generation software recreates the other person's appearance

[1524] The CG generation software recreates the other person's appearance based on the received data. For example, the CG generation software recreates and renders a 3D model of what the other person's face will look like 10 years from now. The input is appearance data, and the output is the recreated 3D model.

[1525] Step 8:

[1526] The server uses the emotion engine to adjust the appearance (if necessary)

[1527] If the user shows a specific emotion, the server uses the emotion engine to adjust the appearance. For example, if the user shows nostalgia, the emotion engine analyzes the emotion and adjusts the appearance to represent a special memorable scene. The input is the user's emotion data, and the output is the adjusted appearance data.

[1528] Step 9:

[1529] The server displays the final look to the user

[1530] The server displays the final generated appearance on the user's device. For example, it displays the other person's face 10 years from now on the user's smartphone. The input is the final appearance data, and the output is the display result on the user's device.

[1531] (Application example 3)

[1532] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1533] Conventional virtual store systems did not allow users to interact with store clerks or customers from the past or future, nor could they adjust the appearance of the other person based on the user's emotions. This limited the user experience and left them with low emotional satisfaction. Furthermore, they were unable to provide the excitement and nostalgia that can be gained through interactions with past and future counterparts.

[1534] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1535] In this invention, the server includes means for understanding the personality of a person the user wishes to meet again, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for meeting the person not only in the past but also in the future, means for adjusting the past appearance of the person in accordance with the user's emotions, and means for interacting with past and future store clerks and customers in the virtual store. This allows the user to interact with past and future store clerks and customers, and by adjusting the appearance of the person in accordance with the user's emotions, it is possible to provide a more moving and nostalgic experience.

[1536] "Means to understand the personality of the person you wish to reunite with" refers to technology that recognizes the characteristics and information of a specific person with whom the user wishes to reunite and acquires this information as data.

[1537] "Means for reproducing realistic interactions based on the image of the other person" is a technology that uses acquired data on the image of the person to simulate natural conversation and communication with that person.

[1538] "Means for visually reproducing the other person" refers to a technology that uses image recognition technology or CG technology to visually reproduce the appearance of the person based on the acquired data of the person's image.

[1539] "A means to meet people not only from the past but also from the future" is a technology that uses AI technology to simulate past and future appearances, allowing users to interact with the person at that time.

[1540] "Means for adjusting the other person's past appearance according to the user's emotions" refers to a technology that recognizes the user's emotional state and adjusts the other person's past appearance accordingly to match a specific emotion.

[1541] "Means for interacting with past and future store clerks and customers in a virtual store" refers to technology that enables users to interact with past and future store clerks and customers in a virtual store.

[1542] The system for implementing this invention includes multiple means for understanding the personality of a person you want to meet again, recreating and visually recreating real interactions, as well as means for meeting past and future counterparts, means for adjusting the past appearance of counterparts according to the user's emotions, and means for interacting with past and future store clerks and customers in a virtual store.

[1543] Hardware and software used

[1544] Hardware: Smartphone, head-mounted display

[1545] Software: OpenCV, DeepFace, EmotionRecognizer, AgeTransformer

[1546] Data processing and calculation

[1547] 1. A way to understand the person you want to reunite with

[1548] The server receives photos and videos of the other person provided by the user and extracts their features using image recognition technology. Specifically, it analyzes the images using OpenCV and recognizes facial features using DeepFace.

[1549] 2. A way to recreate real-life interactions

[1550] The server uses voice recognition technology to imitate the speaking style and tone of the other person, allowing the user to experience a natural conversation. The voice data is analyzed using a voice recognition engine, and data is generated to reproduce the speaking style and tone of the other person.

[1551] 3. Visual Representation

[1552] The server uses CG technology to recreate the appearance of the person based on the acquired facial feature data. Specifically, based on the facial features recognized by DeepFace, it performs age transformation using AgeTransformer and visually recreates the appearance using OpenCV.

[1553] 4. A way to meet past and future partners

[1554] The server adds or subtracts the other person's age based on the user's instructions to generate future or past appearances. For example, if the user requests that they "want to meet the other person 10 years from now," the server uses AgeTransformer to change the other person's appearance 10 years from now.

[1555] 5. A way to adjust the past appearance of the other person depending on the user's emotions

[1556] The server uses EmotionRecognizer to recognize the user's emotions and adjusts the past appearance of the other person according to their emotions. For example, if the user expresses nostalgia, it applies a specific filter to recreate a nostalgic appearance.

[1557] 6. A way to interact with past and future store associates and customers in virtual stores

[1558] The server enables the user to interact with past and future store clerks and customers in the virtual store, allowing the user to enjoy natural interactions with store clerks and customers at a specific point in time in the virtual store.

[1559] Specific examples

[1560] If the user indicates "I want to meet the store clerk from 10 years ago" and expresses nostalgia, the server will display the store clerk's appearance from 10 years ago with a nostalgic filter applied.

[1561] Prompt Sentence Examples

[1562] If the user indicates "I want to meet the store clerk from 10 years ago" and expresses nostalgia, the system should display the store clerk as they were 10 years ago with a nostalgic filter applied.

[1563] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1564] Step 1:

[1565] The user uploads a photo of the person they want to meet again to their device.

[1566] Input: A user-provided photo of the person.

[1567] Operation: The device sends the photo file specified by the user to the server.

[1568] Output: The photo data of the other person sent to the server.

[1569] Step 2:

[1570] The server receives the photo and uses image recognition technology to extract the other person's features.

[1571] Input: The photo data of the other person sent to the server.

[1572] How it works: The server uses OpenCV to analyze the photo and DeepFace to recognize facial features.

[1573] Output: Extracted facial feature data.

[1574] Step 3:

[1575] The server converts the other person's age based on the user's instructions.

[1576] Input: extracted facial feature data, user instructions (e.g., "I want to meet the person I will be in 10 years' time").

[1577] How it works: The server applies an age transformation to facial feature data using the AgeTransformer, changing the appearance by the specified number of years.

[1578] Output: Age-transformed facial feature data.

[1579] Step 4:

[1580] The server recognizes the user's emotions and adjusts the other person's appearance accordingly.

[1581] Input: Age-transformed facial feature data, user emotion data.

[1582] How it works: The server uses EmotionRecognizer to analyze the user's emotions and applies filters to adjust the other person's appearance based on their emotions.

[1583] Output: Refined facial feature data.

[1584] Step 5:

[1585] Based on the adjusted facial feature data, the server uses CG technology to visually recreate the other person's appearance.

[1586] Input: Aligned facial feature data.

[1587] How it works: The server uses CG technology to visually recreate facial feature data and generate an image.

[1588] Output: Reconstructed image of the other person's appearance.

[1589] Step 6:

[1590] The server transmits the reproduced image of the other person's appearance to the terminal and displays it to the user.

[1591] Input: Reconstructed image of the other person's appearance.

[1592] How it works: The server sends the generated image to the device, which displays it to the user.

[1593] Output: An image of the other person's appearance displayed to the user.

[1594] Step 7:

[1595] The user interacts with the person reproduced in the virtual store.

[1596] Input: Appearance image of the other person displayed to the user, voice input.

[1597] How it works: The device sends the user's voice input to a server, which then uses voice recognition technology to mimic the speaking style and tone of the other person, recreating a realistic interaction.

[1598] Output: Interaction data between the user and the reproduced opponent.

[1599] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1600] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1601] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1602] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1603] [Third embodiment]

[1604] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1605] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1606] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1607] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1608] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1609] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1610] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1611] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1612] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1613] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1614] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1615] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1616] "Example 1"

[1617] In one embodiment of the present invention, information provided by the user (e.g., photos, videos, audio, text information, etc.) is collected as a means of understanding the personality of the person the user wishes to reunite with, and an image of the person is generated based on this information using artificial intelligence (AI). The AI ​​learns the person's appearance, speaking style, personality, preferences, etc., and generates an image of the person based on this information.

[1618] "Example 2"

[1619] In addition, as a means of recreating realistic interactions, the AI ​​will converse with the user by imitating the other person's speaking style and tone based on the generated persona. For example, if the user greets the user with "Good morning," the AI ​​will respond with "Good morning" in the other person's tone.

[1620] "Example 3"

[1621] Furthermore, as a means of visually recreating the other person, CG technology is used to mimic the other person's appearance based on the generated image. For example, based on a photo of the other person, AI can recreate their face using CG and display it to the user.

[1622] "Example 4"

[1623] To allow users to meet their partner not only in the past but also in the future, the AI ​​will add or subtract their age and change their appearance. For example, if a user requests to meet their partner in 10 years, the AI ​​will apply the changes to their appearance in 10 years' time and display the results to the user.

[1624] The processing flow of each embodiment will be described below.

[1625] "Example 1"

[1626] Step 1: Collect information from the user about the person they wish to reunite with (e.g., photos, videos, audio, text information, etc.).

[1627] Step 2: Using the collected information, artificial intelligence (AI) is used to generate a profile of the person. The AI ​​learns the person's appearance, speech pattern, personality, preferences, etc.

[1628] "Example 2"

[1629] Step 1: Based on the generated image, the AI ​​imitates the other person's speaking style and tone.

[1630] Step 2: Dialogue with the user. For example, if the user says "Good morning," the AI ​​will respond with "Good morning" in the user's voice.

[1631] "Example 3"

[1632] Step 1: Using the generated image, CG technology is used to mimic the appearance of the other person.

[1633] Step 2: For example, based on a photo of the other person, the AI ​​will recreate their face in CG and display it to the user.

[1634] "Example 4"

[1635] Step 1: The AI ​​adds or subtracts the opponent's age and changes their appearance.

[1636] Step 2: For example, if a user says, "I want to meet my partner 10 years from now," the AI ​​will apply changes to the partner's appearance in 10 years' time and display the results to the user.

[1637] Example 1

[1638] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1639] There is a demand for a system that can provide a detailed understanding of the personality of a person with whom a user wishes to reunite and allow the user to recreate a realistic interaction with that person. However, conventional technologies have had difficulty in comprehensively analyzing the other person's appearance, speech style, personality, preferences, etc. to generate an accurate personality profile. Furthermore, there has been a lack of means to effectively provide the generated personality profile to the user. As a result, users have been unable to obtain detailed information about the person they wish to reunite with, and have been unable to have a satisfying reunion experience.

[1640] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1641] In this invention, the server includes means for collecting information provided by users, means for analyzing the collected information, means for generating a person's profile based on the analyzed information, and means for providing the generated person's profile to the user, thereby enabling the user to accurately understand the detailed profile of the person they wish to reunite with and to recreate a realistic interaction.

[1642] The "means of collecting information provided by users" refers to a function that allows users to upload photos, videos, audio, text information, etc. related to the person they wish to reunite with to the system, which is then received and stored by the server.

[1643] "Means for analyzing collected information" refers to a function that analyzes the photos, videos, audio, and text information collected by the server and extracts characteristics such as the other person's appearance, speaking style, personality, and preferences.

[1644] "Image recognition technology" is a technology for analyzing the external characteristics of others from photographs and videos, and includes algorithms that detect face shape, hairstyle, clothing, etc.

[1645] "Speech recognition technology" is a technology that converts voice data into text and further analyzes the tone and speaking characteristics of the voice.

[1646] "Natural language processing technology" is a technology for analyzing text data and inferring the other person's personality and preferences, and involves the use of generative AI models.

[1647] The "means for generating a person's profile" is a function that creates a detailed profile of the other person by integrating the other person's appearance, speaking style, personality, preferences, etc. based on analyzed information.

[1648] The "means for providing the generated portrait to the user" is a function for notifying the user of the generated portrait on the user's terminal and displaying the profile.

[1649] This invention is a system for enabling a user to understand in detail the personality of a person with whom they wish to reunite and to recreate realistic interactions. This system includes a series of processes that collect and analyze information provided by the user, generate a personality profile of the other person, and provide it to the user.

[1650] First, users upload photos, videos, audio, and text information about the person they want to reunite with to the system. Users can easily provide this information using a smartphone application. For example, users can take photos of the person they want to reunite with and record and upload audio messages.

[1651] The server then collects the information provided by the user and stores it in a database, specifically using cloud storage services such as AWS S3 buckets to store photos and audio files.

[1652] The collected information is analyzed by a server using image recognition technology, such as OpenCV and TensorFlow, to analyze photos and videos and extract the other person's external characteristics, such as facial shape, hairstyle, and clothing.

[1653] Using voice recognition technology, the server converts the voice data into text using voice recognition services such as Google Speech-to-Text API or IBM Watson, and then analyzes the voice tone and speaking characteristics.

[1654] Using natural language processing technology, the server uses generative AI models such as GPT-3 and BERT to analyze text extracted from written and audio data and infer the other person's personality and preferences.

[1655] Based on these analysis results, the server generates a profile of the other person, including their appearance, speaking style, personality, and preferences. For example, it creates a profile that says, "The other person has short hair, speaks softly, and enjoys the outdoors."

[1656] Finally, the server provides the generated profile to the user. Specifically, the server sends a notification to the user's device and displays the generated profile. The user can then open the application and check the detailed profile of the other person.

[1657] An example of a prompt is as follows:

[1658] "You provide a photo and an audio message of the person you want to reunite with. Based on this information, we analyze their appearance, speaking style, personality, and preferences to generate a profile of them."

[1659] In this way, the user can accurately understand the detailed personality of the person they wish to meet again and recreate a realistic interaction.

[1660] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1661] Step 1: The user provides information

[1662] Users upload photos, videos, audio, text information, etc. of the person they want to reunite with to the system. Specifically, users use a smartphone application to take photos of the person they want to reunite with and record and upload audio messages. The input is the photos, videos, audio, and text information provided by the user, and the output is the transmission of this data to the server.

[1663] Step 2: The server collects information

[1664] The server receives the information provided by the user and stores it in a database. Specifically, the server uses cloud storage services such as AWS S3 buckets to store photos and audio files. The input is the photos, videos, audio, and text information sent by the user, and the output is that this data is stored in cloud storage.

[1665] Step 3: The server analyzes the image

[1666] The server uses image recognition libraries such as OpenCV and TensorFlow to analyze the stored photos and videos. Specifically, the server runs a face detection algorithm to extract the facial features of the other person (eye position, nose shape, mouth shape, etc.). The input is the photos and videos stored in cloud storage, and the output is the extracted facial feature data.

[1667] Step 4: The server analyzes the audio

[1668] The server uses speech recognition services such as Google Speech-to-Text API and IBM Watson to analyze the stored voice data. Specifically, the server converts the voice data into text and further extracts voice tone and speaking characteristics. The input is the voice data stored in cloud storage, and the output is text data and voice feature data.

[1669] Step 5: The server parses the text

[1670] The server uses natural language processing models such as GPT-3 and BERT to analyze the text information provided by the user. Specifically, the server analyzes the text data and infers the other person's personality and preferences. The input is the text information stored in cloud storage, and the output is the analyzed personality and preference data.

[1671] Step 6: The server integrates the information and generates a profile

[1672] The server integrates features extracted from images, voice, and text information to generate a profile of the other person. Specifically, the server compiles the other person's appearance, speaking style, personality, and preferences into a single profile. The input is the extracted facial feature data, voice feature data, personality and preference data, and the output is the generated profile of the person.

[1673] Step 7: The server provides the generated profile to the user

[1674] The server provides the generated persona to the user. Specifically, the server sends a notification to the user's device and displays the generated profile. The input is the generated persona profile, and the output is the profile displayed on the user's device.

[1675] (Application example 1)

[1676] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1677] In modern society, it is not easy for people to find someone they want to reunite with. This is especially true in large spaces such as large brick-and-mortar stores and shopping malls. It is also difficult to accurately reproduce the appearance, manner of speech, personality, and other characteristics of the person you want to reunite with and have a realistic interaction. This often leads to missed reunion opportunities, which leads to a decline in user satisfaction.

[1678] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1679] In this invention, the server includes means for understanding the personality of a person the user wishes to reunite with, means for recreating a realistic interaction based on the personality of the person, means for visually recreating the person, means for searching for the person in a physical store, means for notifying the user when the person is found, and means for allowing the user to meet not only past but also future partners. This allows the user to easily find the person they wish to reunite with even in a large physical store, and allows for realistic interactions that accurately reproduce the person's characteristics.

[1680] "A means of understanding the personality of the person you want to reunite with" is a technology in which AI learns the other person's appearance, speaking style, personality, preferences, etc. based on information provided by the user (photos, videos, audio, text information, etc.), and generates an image of that person.

[1681] "Means for recreating realistic interactions" refers to technology that uses voice recognition technology and natural language processing technology to imitate the speaking style and tone of the other person based on a generated image of the other person, and engage in realistic conversations with the user.

[1682] "Means of visual reproduction" refers to technology that uses image recognition technology and computer graphics technology to imitate the appearance of the other person and visually reproduce the other person for the user.

[1683] The "means of finding a partner within a physical store" refers to a technology that analyzes camera footage within a physical store in real time and uses facial recognition technology to find the person you want to meet again with.

[1684] The "means for notifying when the person is found" refers to a technology that sends a notification to a device such as smart glasses or a smartphone when a person you want to reunite with is found, informing the user.

[1685] "A means to meet not only past but also future partners" is a technology that allows you to save information about not only past partners you want to meet again with, but also people you want to meet again with in the future, and then reunite with those people in the future.

[1686] The system for implementing this invention has the function of understanding the profile of the person the user wants to meet, searching for that person in a physical store, and notifying the user when the person is found. A specific embodiment of this system will be described below.

[1687] First, the user provides information about the person they want to reunite with. This information includes photos, videos, audio, and text of the person. This information is sent to a server and used as training data for an AI model. The server uses facial recognition libraries such as DeepFace and face_recognition to learn the person's appearance, speaking style, personality, preferences, etc., and generates a profile of that person.

[1688] Next, to find the person in the physical store, the server analyzes the camera footage in the physical store in real time. It uses image processing libraries such as OpenCV to detect faces from the camera footage and identifies the person using facial recognition technology. The server then matches the detected face with an AI model to find the person you want to reunite with.

[1689] When a match is found, the server sends a notification to the device, such as smart glasses or a smartphone. The notification informs the user that the match has been found and offers an opportunity to reunite. The notification is displayed on the smart glasses display or the smartphone's notification function.

[1690] For example, a user enters a photo and characteristics of a person they would like to meet in a shopping mall into the smart glasses. The server trains an AI model based on the provided information and analyzes camera footage in the mall to find the person. When the person is found, the smart glasses display a notification saying "Partner Found!"

[1691] Example prompt sentence:

[1692] Develop an application that allows users to provide photos, videos, audio, and text information about the person they wish to reunite with, and uses AI to generate a profile of that person. The application will be installed on smart glasses and will help users find their match in a physical store. Specifically, the AI ​​will generate a profile of the person based on the information provided by the user, and then locate that person in the store in real time.

[1693] In this way, users can easily find the person they want to meet again even in a large physical store, and can have a realistic interaction that accurately reproduces the other person's characteristics.

[1694] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1695] Step 1:

[1696] The user provides information about the person they want to meet. The user inputs photos, videos, audio, and text information about the other person into their device. This information is then sent to the server. The input data includes characteristics such as the other person's appearance, speaking style, personality, and preferences.

[1697] Step 2:

[1698] The server trains an AI model based on the information provided. The server uses facial recognition libraries such as DeepFace and face_recognition to learn the other person's appearance. It also uses voice recognition technology to learn the other person's speaking style and tone. This generates a profile of the other person. The input data is the information provided by the user, and the output data is the trained AI model.

[1699] Step 3:

[1700] The server analyzes camera footage from inside the physical store in real time. The server uses image processing libraries such as OpenCV to detect faces from the camera footage. The detected faces are compared with an AI model. The input data is the camera footage from inside the physical store, and the output data is the location information of the detected faces.

[1701] Step 4:

[1702] The server uses facial recognition technology to identify the person you want to reunite with. The server then matches the det...

Claims

1. A means for acquiring an image of a person to be reunited, the image being uploaded by the user; A means for generating computer graphics that reproduce the face of the other party using a prompt sentence that instructs the other party to reproduce the face of the other party in computer graphics based on the image of the other party and a generative AI model; A means for accepting an instruction inputted by a user that the user wishes to meet a future or past contact; a means for modifying the generated computer graphics using a prompt sentence instructing to add an age change to the generated computer graphics based on the received instruction and the generative AI model; means for displaying the modified computer graphics; A system including:

2. A means for acquiring voice data of the person to be reunited; A means for analyzing the speaking style or tone of the other party from the voice data of the other party; A means for reproducing a conversation between the user and the other party using a prompt sentence that instructs the user to converse with the other party based on an input from the user and an analysis result of the other party's speaking style or tone, and the generative AI model; a means for outputting the utterance of the other person in the reproduced conversation; The system of claim 1 further comprising:

3. means for recognizing an emotion of the user in the conversation using an emotion engine; The system of claim 2, wherein the reproduction means reproduces a conversation between the user and the other party using the prompt sentence instructing the other party to converse with the user based on input from the user, the user's emotions, and an analysis result of the other party's speaking style or tone, and the generative AI model.

Images