System

Abstract

A system is provided.SOLUTION: A data processing apparatus, comprising: means for receiving authentication information input by a user; means for querying a database for the authentication information to obtain an authentication result; means for displaying a home screen or an error message of the user based on the authentication result; means for performing voice recognition on input data from the user and converting the input data into text data; means for analyzing the text data to generate an appropriate response; means for generating deepfake video and audio data based on the response; and means for transmitting and playing the generated video and audio data to a user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the recent evolution of deepfake technology, the risks of its misuse are increasing. Issues such as privacy violations and the spread of fake news are particularly serious. Meanwhile, there is a need to utilize deepfake technology and voice bots for sound purposes and improve the quality of entertainment. This invention aims to promote the sound use of deepfake technology by providing a system that allows users to enjoy two-way interactions with their "oshi" (favorite idols) through official applications exclusive to fan clubs and online salons. Furthermore, it is necessary to thoroughly protect privacy and copyright and prevent the misuse of the technology. [Means for solving the problem]

[0005] The present invention is a system that includes the following means. First, a means is provided for receiving authentication information entered by a user and querying a database to obtain an authentication result. Based on the authentication result, a home screen or an error message for the user is displayed. Next, a means is provided for speech recognition of data input by the user and converting it into text data. A natural language processing means is provided for analyzing the converted text data and generating a response. Finally, a means is provided for generating deepfake video and audio data based on the response and transmitting them to the user terminal for playback.

[0006] Furthermore, a means is provided for receiving an alarm time and message set by a user, generating deepfake video and audio data based on the alarm time, and playing the data on the device at the alarm time. Also, a means is provided for receiving song request information entered by a user, verifying copyright information, generating deepfake video and audio data, and transmitting the data to the user's device for playback, thereby building a system that thoroughly protects copyright and privacy.

[0007] A "data processing device" is a device that receives, processes, and transmits user-provided information.

[0008] "User" refers to a person who uses this system.

[0009] "Authentication information" refers to information required for authentication, such as ID and password entered by the user.

[0010] "Database" refers to a system for storing and managing user information, alarm settings, request information, etc.

[0011] "Home screen" refers to the main screen that is displayed when user authentication is successful.

[0012] "Error message" refers to a message that displays a warning or notification when authentication fails.

[0013] "Input Data" refers to information provided by a user through speech or text.

[0014] "Speech recognition" is a technology that converts user-provided voice data into text data.

[0015] "Text data" refers to text information generated by speech recognition.

[0016] "Natural language processing means" refers to technology for analyzing text data and generating appropriate responses.

[0017] "Deepfake video" refers to synthetic video generated using AI technology.

[0018] "Audio data" refers to audio information generated by multimodal generation AI.

[0019] "User terminal" refers to a device (smartphone, tablet, PC, etc.) used by a user to access the System.

[0020] "Alarm time" refers to the wake-up time or notification time set by the user.

[0021] "Message" refers to the text information that is displayed and played back when an alarm or request is made by the user.

[0022] "Request information" refers to information provided by a user when requesting a particular piece of content, such as a song.

[0023] "Copyright Information" refers to data used to verify whether particular content is copyrighted.

[0024] A "natural language processing engine" refers to software or hardware that analyzes text data entered by a user and generates a response.

[0025] An "HTTP request" refers to a communication protocol for sending data from a client (user terminal) to a server.

[0026] "Response" refers to response data sent from the server to the client (user terminal). [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0028] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0029] First, the terms used in the following description will be explained.

[0030] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0031] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0032] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0033] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0034] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0035] [First embodiment]

[0036] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0037] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0038] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0039] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0040] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0041] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0042] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0043] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0044] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0045] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0046] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0047] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] This invention is a system that allows users to enjoy interactive communication with their "oshi" using a data processing device. Specifically, it utilizes multimodal generation AI with deepfake technology and voice bots, while taking privacy and copyright protection into consideration.

[0049] System Configuration

[0050] This system consists of a user terminal, a server, a database, and an AI engine (natural language processing engine, deepfake generation engine, and voice synthesis engine). The specific operation of each element is explained below.

[0051] User Authentication

[0052] 1. The user launches the app and enters their login information.

[0053] 2. The device sends the login information to the server.

[0054] 3. The server queries the database for login information and obtains the authentication result.

[0055] 4. The server sends the user's home screen if authentication is successful, and an error message if authentication fails.

[0056] 5. The device displays the home screen or an error message.

[0057] Conversation feature

[0058] 1. The user selects the conversation feature and enters information by voice or text.

[0059] 2. The device converts voice input into text (when using voice input).

[0060] 3. The terminal sends the entered text data to the server.

[0061] 4. The server analyzes the text data using a natural language processing engine and generates an appropriate response.

[0062] 5. The server generates response video and audio using a deepfake generation engine and a voice synthesis engine.

[0063] 6. The server transmits the generated video and audio data to the user terminal.

[0064] 7. The device plays video and audio, allowing the user to have a conversation with their favorite idol.

[0065] Alarm settings

[0066] 1. The user sets the alarm time and message.

[0067] 2. The device sends the configuration information to the server.

[0068] 3. The server stores the configuration information in a database.

[0069] 4. When the alarm time arrives, the server generates deepfake video and audio and sends them to the user's device.

[0070] 5. The device will play an alarm along with video and audio, and your favorite idol will wake you up.

[0071] Song Requests

[0072] 1. The user enters song request information.

[0073] 2. The device sends the request information to the server.

[0074] 3. The server checks the copyright information for the requested song.

[0075] 4. If copyright is permitted, the server generates deepfake video and audio and sends them to the user's device.

[0076] 5. The device plays the video and audio, and the user enjoys watching their favorite idol sing the song.

[0077] Specific examples

[0078] Example 1: Conversation function

[0079] When a user launches the app and logs in, the home screen appears.

[0080] The user taps the conversation icon and says, "Hello, how are you today?"

[0081] The device converts the speech into text and sends it to the server.

[0082] The server generates a response saying, "How are you today? How are you?", creating deepfake video and audio.

[0083] When the device plays the video and audio, your favorite idol will say, "I'm feeling good today. How about you?"

[0084] Example 2: Alarm settings

[0085] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[0086] When the alarm time comes, the server generates video and audio saying "Good morning, it's time to wake up" and sends it to the device.

[0087] The device will sound an alarm while playing video and audio, and the "oshi" will wake the user up.

[0088] The system of the present invention allows users to enjoy two-way communication with their "oshi" (favorite idols). It also thoroughly protects privacy and copyrights, promoting sound use.

[0089] The processing flow will be explained below.

[0090] Conversation function processing steps

[0091] Step 1:

[0092] The user launches the application and the login screen appears.

[0093] Step 2:

[0094] The user enters login information (username, password) and taps the login button.

[0095] Step 3:

[0096] The terminal sends the entered login information to the server.

[0097] Step 4:

[0098] The server queries the database for login information and authenticates the user.

[0099] Step 5:

[0100] The server returns the authentication result to the device. If authentication is successful, it returns the home screen data, and if it is unsuccessful, it returns an error message.

[0101] Step 6:

[0102] The device displays the home screen or an error message.

[0103] Step 7:

[0104] The user taps the conversation feature icon on the home screen and enters a message by voice or text.

[0105] Step 8:

[0106] The device calls the speech recognition API to convert voice input into text (when voice input is performed).

[0107] Step 9:

[0108] The terminal transmits the text data to the server.

[0109] Step 10:

[0110] The server passes the text data to a natural language processing engine and begins analysis.

[0111] Step 11:

[0112] A natural language processing engine generates appropriate response text.

[0113] Step 12:

[0114] The server passes the response text to the deepfake generation engine and speech synthesis engine.

[0115] Step 13:

[0116] The deepfake generation engine generates the video response, and the speech synthesis engine generates the audio response.

[0117] Step 14:

[0118] The server transmits the generated video and audio data to the terminal.

[0119] Step 15:

[0120] The device plays back the received video and audio data, enabling conversation with the "favorite."

[0121] Alarm configuration process steps

[0122] Step 1:

[0123] The user selects the alarm setting function and displays the setting screen.

[0124] Step 2:

[0125] The user inputs the alarm time and the message they want to wake up with.

[0126] Step 3:

[0127] The terminal sends the set alarm time and message to the server.

[0128] Step 4:

[0129] The server stores the alarm configuration information in a database.

[0130] Step 5:

[0131] When the alarm time arrives, the server retrieves the saved setting data.

[0132] Step 6:

[0133] The server generates deepfake video and audio data for the alarm.

[0134] Step 7:

[0135] The server transmits the generated data to the terminal.

[0136] Step 8:

[0137] The device will sound an alarm and play deepfake video and audio.

[0138] Song request processing steps

[0139] Step 1:

[0140] The user selects the song request function and the request screen is displayed.

[0141] Step 2:

[0142] The user inputs the title of the song they wish to request.

[0143] Step 3:

[0144] The terminal sends the request information to the server.

[0145] Step 4:

[0146] The server checks the copyright information of the requested song against its database.

[0147] Step 5:

[0148] If copyright permits, the server generates the deepfake video and audio data.

[0149] Step 6:

[0150] The server transmits the generated data to the terminal.

[0151] Step 7:

[0152] The device plays back the received video and audio data and provides a video of the "favorite" person singing the song.

[0153] Example 1

[0154] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0155] With current information processing technology, it is difficult for users to enjoy interactive communication with virtual characters in real time. Furthermore, there is a lack of systems that can generate high-quality audio and video while fully considering privacy and copyright protection. This limits the user experience and reduces satisfaction. Furthermore, implementing features such as alarms and song requests is complex, creating a demand for more user-friendly systems.

[0156] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0157] In this invention, the server includes means for receiving authentication information entered by a user, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting the user's input data into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for generating deepfake video and audio data based on the response, means for transmitting the generated video and audio data to a user terminal and playing it back, means for encrypting the data using SSL / TLS, means for understanding the context and generating a response using a natural language processing engine, and means for generating deepfake technology in high resolution and in real time. This allows users to enjoy high-quality two-way communication in real time through audio and video, and also allows them to use the system safely while protecting their privacy and copyright.

[0158] "User" refers to an individual or organization that uses the system.

[0159] "Authentication information" is information for verifying the identity of a user, and includes a username and password.

[0160] A "database" is a system for efficiently storing and managing information.

[0161] "Home screen" refers to the main operation screen that is displayed after a user has successfully logged in.

[0162] An "error message" is a warning or explanatory message that is sent to the user when an error occurs in the system.

[0163] "Speech recognition" is a technology that converts input speech into text data.

[0164] "Text data" is data that is stored or processed as textual information.

[0165] "Natural language processing" is a technology that allows computers to understand and analyze human language.

[0166] "Deepfake" is a technology that uses artificial intelligence to generate or alter video and audio.

[0167] "Audio data" is data that is stored or processed as audio information.

[0168] "Transmitting" is the act of transferring data from one device to another.

[0169] "Playback" refers to the act of outputting stored audio or video in a form that can be confirmed visually or audibly.

[0170] "Encryption" is a technology that converts data according to rules to make it unreadable to third parties.

[0171] An "alarm" is a function that notifies or warns the user at a specific time set by the user.

[0172] "Copyright" refers to the legal right to protect creative works.

[0173] "Request" means a request from a User to take a particular action or provide data.

[0174] The present invention is a system that allows a user to enjoy two-way communication with a virtual character using a data processing device. Specific embodiments of the present invention will be described below.

[0175] System Configuration

[0176] This system consists of a user terminal, a server, a database, and an AI engine (natural language processing engine, deepfake generation engine, and voice synthesis engine).

[0177] User Authentication

[0178] A user launches the app and enters their login information. The device sends the entered login information to the server. The server queries the database for the login information and obtains the authentication result. If authentication is successful, the server generates a home screen and sends it to the device. If authentication fails, the server generates an error message and sends it to the device. The device displays the home screen or the error message.

[0179] Conversation feature

[0180] The user selects the conversation function and inputs voice or text. The device converts the voice input into text (in the case of voice input). The device sends the converted text data to the server. The server analyzes it using a natural language processing engine and generates an appropriate response. The server creates a response video and audio using a deepfake generation engine and a speech synthesis engine. The server sends the generated video and audio data to the user's device. The device plays the video and audio, and the user enjoys a conversation with their "favorite character."

[0181] Alarm settings

[0182] The user opens the alarm settings and enters the alarm time and message to set it. The device sends the setting information to the server. The server saves the setting information in a database. When the alarm time arrives, the server generates deepfake video and audio and sends it to the user's device. The device plays the video and audio along with the alarm time, and the "oshi" wakes the user up.

[0183] Song Requests

[0184] The user inputs song request information on the app. The device sends the request information to the server. The server checks the copyright information of the requested song in a database. If the copyright is permitted, the server generates deepfake video and audio and sends it to the user's device. The device plays the video and audio, allowing the user to enjoy watching their favorite idol sing.

[0185] Hardware and software used

[0186] User devices: smartphones, tablets, PCs

[0187] Server: High-performance computer, cloud server

[0188] Database: SQL database, NoSQL database

[0189] AI engines: natural language processing engines (e.g., BERT), deepfake generation engines, speech synthesis engines (e.g., Tacotron)

[0190] Specific examples

[0191] Example 1: Conversation function

[0192] When a user launches the app and logs in, the home screen appears.

[0193] The user taps the conversation icon and says, "Hello, how are you today?"

[0194] The device converts the speech into text and sends it to the server.

[0195] The server generates a response saying, "How are you today? How are you?", creating deepfake video and audio.

[0196] When the device plays the video and audio, your favorite idol will say, "I'm feeling good today. How about you?"

[0197] Example 2: Alarm settings

[0198] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[0199] When the alarm time comes, the server generates video and audio saying "Good morning, it's time to wake up" and sends it to the device.

[0200] The device will sound an alarm while playing video and audio, and the "oshi" will wake the user up.

[0201] Prompt Sentence Examples

[0202] 1. "What happens when a user selects the conversation feature and types, 'Hello, how are you today?'"

[0203] 2. "What happens when a user sets an alarm for 7:00 and is woken up by a specified message?"

[0204] The system of the present invention allows users to enjoy two-way communication with their "oshi" and ensures thorough privacy and copyright protection.

[0205] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0206] User authentication process steps

[0207] Step 1:

[0208] The user launches the app and enters their username and password on the login screen.

[0209] Input: The username and password entered by the user

[0210] Output: Login information prepared to be sent to the terminal

[0211] Specific behavior: A login form is displayed on the user interface.

[0212] Step 2:

[0213] The terminal sends the entered username and password to the server.

[0214] Input: Login information (username and password)

[0215] Output: Login information sent to the server

[0216] Specific operation: Login information is encrypted with SSL / TLS and sent to the server.

[0217] Step 3:

[0218] The server queries the database for the received login information and searches for corresponding user data.

[0219] Input: Login information sent to the server

[0220] Output: Authentication result from the database

[0221] What happens: A search query is executed against the database using the username and password.

[0222] Step 4:

[0223] The server determines whether the authentication was successful or not based on the results obtained from the database.

[0224] Input: Authentication results from the database

[0225] Output: Home screen data or error message

[0226] Specific operation: If authentication is successful, home screen data based on the user information is generated; if authentication fails, an error message is generated.

[0227] Step 5:

[0228] The terminal displays the home screen data or an error message received from the server.

[0229] Input: Home screen data or error message sent from the server

[0230] Output: The home screen or error message that the user sees

[0231] Specific Behavior: The display screen is updated and an appropriate message is displayed to the user.

[0232] Conversation function processing steps

[0233] Step 1:

[0234] The user selects the conversation function and enters information by voice or text.

[0235] Input: Voice or text input from the user

[0236] Output: Input data received on the terminal

[0237] Specific behavior: The conversation icon is tapped, and an interface for voice or text input appears.

[0238] Step 2:

[0239] The device converts the input voice into text.

[0240] Input: Audio data (in the case of audio input)

[0241] Output: Text data

[0242] Specific operation: The voice recognition engine analyzes the voice data and converts it into text data.

[0243] Step 3:

[0244] The terminal transmits the converted text data to the server.

[0245] Input: Text data (including voice input)

[0246] Output: Text data sent to the server

[0247] Specific operation: The text data is encrypted and sent to the server.

[0248] Step 4:

[0249] The server analyzes the received data using a natural language processing engine and generates an appropriate response.

[0250] Input: Text data sent to the server

[0251] Output: Response text data

[0252] What it does: A natural language processing engine analyzes text data and generates contextual responses.

[0253] Step 5:

[0254] Based on the response generated by the server, video and audio are generated using a deepfake generation engine and a voice synthesis engine.

[0255] Input: Response text data

[0256] Output: Response video and audio data

[0257] Specific operation: The deepfake generation engine generates a response video, and the speech synthesis engine generates a response voice.

[0258] Step 6:

[0259] The server transmits the generated video and audio data to the user terminal.

[0260] Input: Response video and audio data

[0261] Output: Video and audio data sent to the user's device

[0262] What it does: Data is encrypted and efficiently transmitted to the device.

[0263] Step 7:

[0264] The device plays back the video and audio received.

[0265] Input: Video and audio data sent from the server

[0266] Output: The video and audio played to the user

[0267] Specific operation: Video and audio are played, and the user enjoys a conversation with their "favorite."

[0268] Alarm configuration process steps

[0269] Step 1:

[0270] The user opens the alarm settings and sets the alarm time and message.

[0271] Input: Alarm time and message

[0272] Output: Configuration information received on the device

[0273] Specific operation: The alarm setting screen is displayed and the user enters the time and message.

[0274] Step 2:

[0275] The terminal sends the setting information to the server.

[0276] Input: Alarm time and message

[0277] Output: Configuration information sent to the server

[0278] Specific operation: The configuration information is encrypted and sent to the server.

[0279] Step 3:

[0280] The server stores the configuration information in a database.

[0281] Input: Configuration information sent to the server

[0282] Output: Configuration information stored in the database

[0283] Specific behavior: High availability settings are made when data is saved to the database.

[0284] Step 4:

[0285] When the alarm time arrives, the server generates deepfake video and audio and sends them to the user's device.

[0286] Input: Saved alarm time and message

[0287] Output: Generated deepfake video and audio data

[0288] Specific operation: The alarm is triggered at the alarm time, and video and audio are generated and transmitted.

[0289] Step 5:

[0290] When the alarm time comes on the device, it plays video and audio.

[0291] Input: Video and audio data sent from the server

[0292] Output: The video and audio played to the user

[0293] Specific operation: Video and audio are played in sync with the alarm sound, and the "oshi" character wakes up the user.

[0294] Song request processing steps

[0295] Step 1:

[0296] The user enters song request information on the app.

[0297] Input: Song request information

[0298] Output: Request information received on the device

[0299] Specific operation: A request screen is displayed, and the user enters the desired song title and artist name.

[0300] Step 2:

[0301] The terminal sends the request information to the server.

[0302] Input: Request information

[0303] Output: Request information sent to the server

[0304] Specific operation: The request information is encrypted and sent to the server.

[0305] Step 3:

[0306] The server checks the copyright information of the requested song against its database.

[0307] Input: Request information sent to the server

[0308] Output: Copyright information check result

[0309] What it does: Query the database for copyright information for the requested song.

[0310] Step 4:

[0311] If copyright is permitted, the server generates deepfake video and audio and sends them to the user's device.

[0312] Input: Copyright permission confirmation result, requested song information

[0313] Output: Generated deepfake video and audio data

[0314] Specific operation: The generated video and audio are encrypted and sent to the user terminal.

[0315] Step 5:

[0316] The device plays the video and audio, and the user enjoys watching their favorite idol sing.

[0317] Input: Video and audio data sent from the server

[0318] Output: The video and audio played to the user

[0319] Specific operation: The playback screen is displayed and video and audio are played.

[0320] This concludes the detailed explanation of each processing step in this system, which allows users to enjoy high-quality two-way communication in real time.

[0321] (Application example 1)

[0322] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] Conventional advertising systems have difficulty providing interactive and effective advertisements tailored to users' interests. There is a particular demand for advertising that utilizes "favorites" that users feel attached to, but current technology does not provide an effective means for achieving this. Furthermore, if real-time responses to user input are required, advanced technology is required to achieve this. An effective system is needed to solve these problems.

[0324] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0325] In this invention, the server includes a means for a user to select an advertisement display, a means for generating advertising video and audio data featuring the user's "oshi" using a deepfake generation engine and a voice bot, a means for delivering and playing the generated advertising video and audio data to the user's terminal, a means for receiving questions and comments from the user, analyzing them with natural language processing means, and generating appropriate responses, and a means for playing the generated responses via the user's "oshi" using the natural language processing means and the voice bot. This allows users to receive responses in real time while enjoying interactive advertising via their "oshi."

[0326] The "means for receiving authentication information entered by a user" refers to a component or system for receiving login information and authentication information entered by a user via a network.

[0327] The "means for querying a database for authentication information and obtaining an authentication result" refers to a process or system for querying a database for received authentication information, verifying its validity, and obtaining an authentication result.

[0328] "Means for displaying the user's home screen or an error message based on the authentication result" refers to an interface or software for displaying an appropriate screen (home screen or an error message) on the user's device depending on the authentication result.

[0329] "Means for recognizing voice input data from a user and converting it into text data" refers to technology or software that converts voice input data into text format.

[0330] The "natural language processing means for analyzing text data and generating an appropriate response" refers to a natural language processing engine or algorithm for analyzing text data and generating an appropriate response.

[0331] A "means for generating deepfake video and audio data based on responses" is a process or system that uses deepfake technology to create video and audio data based on generated responses.

[0332] "Means for transmitting the generated video and audio data to a user terminal and playing it" refers to a device or application for transmitting the generated deepfake video and audio data to a user's device via a network and playing it.

[0333] The "means by which a user can select to display an advertisement" refers to an interface or software that allows a user to select to view an advertisement.

[0334] "Means for generating advertising video and audio data featuring a user's favorite character using a deepfake generation engine and voice bot" refers to an engine or system that makes full use of deepfake technology and voice bots to create advertising video and audio featuring a user's favorite character or idol.

[0335] "Means for delivering and playing the generated advertising video and audio data to a user terminal" refers to a method or apparatus for transmitting the generated advertising video and audio data to a user's device via a network and automatically playing them.

[0336] "Means for receiving questions and comments from users, analyzing them using natural language processing means, and generating appropriate responses" refers to natural language processing technology for receiving questions and comments from users, analyzing them, and creating appropriate responses.

[0337] "Means for using the natural language processing means and a voice bot to play back the generated response through the user's "oshi"" refers to a system or method for using a voice bot to play back a response generated by natural language processing in the voice of the user's "oshi."

[0338] This invention is an interactive advertising display system that uses the user's "oshi" (favorites) and utilizes deepfake technology and voice bots to provide advertisements featuring people special to the user. The following describes how this system can be specifically realized.

[0339] System Configuration

[0340] The system consists of the following main components:

[0341] 1. User device: Mobile device such as a smartphone (iOS / ANDROID (registered trademark))

[0342] 2. Server: A cloud-based server (e.g., AWS EC2 instance)

[0343] 3. Database: A database for data storage and authentication (e.g., MySQL (registered trademark) on AWS RDS)

[0344] 4. AI engines: Deepfake generation engines, voice bots, natural language processing engines (e.g., Google® Cloud NLP API, TENSORFLOW®, Twilio Voice API)

[0345] User Authentication

[0346] 1. The user launches the app and enters their login information.

[0347] 2. The device sends the login information to the cloud server.

[0348] 3. The server queries the database for login information and obtains the authentication result.

[0349] 4. If the server is successful, it sends the user's home screen; if not, it sends an error message.

[0350] 5. The device displays the home screen or an error message.

[0351] Ad display function

[0352] 1. The user clicks (or taps) the button to view the ad.

[0353] 2. The server uses a deepfake generation engine and a voice bot to generate advertising video and audio data featuring the user's "favorite person."

[0354] 3. The server distributes the generated video and audio data to the user terminal.

[0355] 4. The device plays the advertising video and audio, and the "oshi" delivers the advertisement to the user.

[0356] Interactive Features

[0357] 1. Users can type in questions or comments during the ad.

[0358] 2. The terminal sends the user's input to the server.

[0359] 3. The server analyzes the user's input using a natural language processing engine and generates an appropriate response.

[0360] 4. The server uses deepfakes and a voice bot to generate video and audio data of the "favorite" responding.

[0361] 5. The terminal plays a video and audio response to the user.

[0362] Recommendations

[0363] 1. The server analyzes the user's past behavior data in the database.

[0364] 2. The server selects the most suitable advertisement based on the analysis results.

[0365] 3. The server generates the selected advertisement using a deepfake generation engine and a voice bot and delivers it to the user's device.

[0366] Specific prompt examples

[0367] A prompt that the user types in the app: "How do I use this product?"

[0368] Prompt the server-side AI model: "Please explain in the best possible terms how to use the product the user cares about."

[0369] In this way, the system of the present invention provides a technique that allows for real-time responses while providing interactive advertising via the user's "oshi."

[0370] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0371] Step 1:

[0372] A user launches the app and enters their login information, which includes their user ID and password, and is sent to the device.

[0373] Step 2:

[0374] The terminal sends the received login information to the cloud server. Specifically, the login information is sent to the cloud server using the HTTPS protocol.

[0375] Step 3:

[0376] The server receives the login information and queries a database, which stores the user's authentication information, to verify its validity.

[0377] Step 4:

[0378] The server obtains the authentication result, generates the user's home screen data if successful, or generates an error message if unsuccessful, and sends this information to the device.

[0379] Step 5:

[0380] The device displays the home screen or an error message, using an HTML rendering engine to display the appropriate screen.

[0381] Step 6:

[0382] The user selects to display an ad. A button click is registered as input, and the device sends this information to the server.

[0383] Step 7:

[0384] The server receives the button click information and launches the deepfake generation engine and voice bot. The deepfake generation engine generates video using TensorFlow, and the voice bot generates audio using the Twilio API.

[0385] Step 8:

[0386] The server transmits the generated advertising video and audio data to the user terminal, where the data is transmitted in compressed form and decoded.

[0387] Step 9:

[0388] The device plays the advertisement video and audio, using the device's video and audio players.

[0389] Step 10:

[0390] The user enters questions or comments during the advertisement. The input is text data, which is sent from the terminal to the server.

[0391] Step 11:

[0392] The server receives the user's input and analyzes it using a natural language processing engine (Google Cloud NLP API), where the input text is parsed and the appropriate response data is generated.

[0393] Step 12:

[0394] The server processes the response data using a deepfake generation engine and voice bot, generating video and audio data in which the "favorite" responds.

[0395] Step 13:

[0396] The server transmits the generated response video and audio data to the user terminal, where the data is recompressed and transmitted.

[0397] Step 14:

[0398] The device plays back the response video and audio, allowing the user to receive a response from their "oshi" in real time.

[0399] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0400] System Configuration

[0401] This invention is a system for realizing two-way communication with users using a data processing device. Specifically, it utilizes deepfake technology, a voice bot, and a multimodal generation AI using an emotion engine to realize real-time interaction with users.

[0402] Basic components

[0403] The system consists of the following main components:

[0404] User devices: smartphones, tablets, PCs, etc.

[0405] Server: Data processing and generation processing

[0406] Database: Stores user information, alarm settings, request information, etc.

[0407] AI engines: natural language processing engine, deepfake generation engine, speech synthesis engine, emotion engine

[0408] User Authentication

[0409] 1. The user launches the app and enters their login information (username, password).

[0410] 2. The device sends the input information to the server.

[0411] 3. The server queries the database for login information and obtains the authentication result.

[0412] 4. The server returns the authentication result to the device, and if authentication is successful, the home screen is displayed, and if it is unsuccessful, an error message is displayed.

[0413] 5. The device displays the home screen or an error message.

[0414] Conversation function using emotion recognition

[0415] 1. The user selects the conversation feature and enters information by voice or text.

[0416] 2. The device uses a speech recognition API to convert voice input into text (when voice input is used).

[0417] 3. The terminal sends the entered text data to the server.

[0418] 4. The server passes the text data to the natural language processing engine and begins analysis.

[0419] 5. The natural language processing engine generates the appropriate response text.

[0420] 6. The server uses the emotion engine to recognize emotions from the user's input data.

[0421] 7. The emotional information recognized by the emotion engine is reflected in response generation.

[0422] 8. The server generates a response video and audio using a deepfake generation engine and a voice synthesis engine, adjusting the facial expression and tone of the audio based on the recognized user emotion.

[0423] 9. The server sends the generated video and audio data to the terminal.

[0424] 10. The device plays video and audio, allowing the user to have a conversation with their favorite idol.

[0425] Alarm setting using emotion recognition

[0426] 1. The user sets the alarm time and message.

[0427] 2. The device sends the configuration information to the server.

[0428] 3. The server stores the configuration information in a database.

[0429] 4. When the alarm time arrives, the server retrieves the data and uses an emotion engine to generate deepfake video and audio data that matches the user's emotions.

[0430] 5. The server sends the generated data to the device.

[0431] 6. The device sounds an alarm and plays emotionally appropriate video and audio to wake the user up.

[0432] Song Request Using Emotion Recognition

[0433] 1. The user enters song request information.

[0434] 2. The device sends the request information to the server.

[0435] 3. The server checks the copyright information of the requested song against its database.

[0436] 4. If copyright permits, the server uses an emotion engine to generate deepfake video and audio data based on the user's emotions.

[0437] 5. The server sends the generated data to the device.

[0438] 6. The device plays the video and audio data, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[0439] Specific examples

[0440] Example 1: Conversation function

[0441] When a user launches the app and logs in, the home screen appears.

[0442] The user taps the conversation icon and says, "Hello, how are you today?"

[0443] The device converts the speech into text and sends it to the server.

[0444] The server performs text analysis and generates a response: "I'm fine today, how about you?"

[0445] The emotion engine recognizes emotions from the user's voice and reflects them in the response.

[0446] The deepfake generation engine generates images that match the user's emotions, and the voice synthesis engine adjusts the tone to generate voice.

[0447] The device plays video and audio, and your favorite idol speaks to you, saying, "I'm feeling good today. How about you?"

[0448] Example 2: Alarm settings

[0449] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[0450] When the alarm time comes, the server receives the message "Good morning, it's time to wake up" and uses an emotion engine to generate video and audio that matches the user's emotions.

[0451] The device sounds an alarm and plays video and audio that matches the user's emotions, and the ``oshi'' character wakes the user up.

[0452] The system of the present invention allows users to enjoy real-time interactive communication with their "oshi" and provides a personalized experience based on their emotions. It also thoroughly protects privacy and copyrights and promotes the healthy use of technology.

[0453] The processing flow will be explained below.

[0454] Processing steps for conversational functions using emotion recognition

[0455] Step 1:

[0456] The user launches the application and the login screen appears.

[0457] Step 2:

[0458] The user enters login information (username, password) and taps the login button.

[0459] Step 3:

[0460] The terminal sends the entered login information to the server.

[0461] Step 4:

[0462] The server queries the database for login information and authenticates the user.

[0463] Step 5:

[0464] The server returns the authentication result to the device. If authentication is successful, it returns the home screen data, and if it is unsuccessful, it returns an error message.

[0465] Step 6:

[0466] The device displays the home screen or an error message.

[0467] Step 7:

[0468] The user taps the conversation feature icon and enters a message by voice or text.

[0469] Step 8:

[0470] The device calls the speech recognition API to convert voice input into text (when voice input is performed).

[0471] Step 9:

[0472] The terminal transmits the text data to the server.

[0473] Step 10:

[0474] The server passes the text data to a natural language processing engine and begins analysis.

[0475] Step 11:

[0476] A natural language processing engine generates appropriate response text.

[0477] Step 12:

[0478] The server uses an emotion engine to recognize emotions from the user's input data.

[0479] Step 13:

[0480] The emotional information recognized by the emotion engine is reflected in response generation.

[0481] Step 14:

[0482] The server uses a deepfake generation engine and a voice synthesis engine to generate a response video and audio, adjusting the facial expression and tone of the audio based on the user's recognized emotions.

[0483] Step 15:

[0484] The server transmits the generated video and audio data to the terminal.

[0485] Step 16:

[0486] The device plays back the received video and audio data, enabling conversation with the "favorite."

[0487] Processing steps for setting alarms using emotion recognition

[0488] Step 1:

[0489] The user selects the alarm setting function and displays the setting screen.

[0490] Step 2:

[0491] The user inputs the alarm time and the message they want to wake up with.

[0492] Step 3:

[0493] The terminal sends the set alarm time and message to the server.

[0494] Step 4:

[0495] The server stores the alarm configuration information in a database.

[0496] Step 5:

[0497] When the alarm time arrives, the server acquires the set alarm information.

[0498] Step 6:

[0499] The server uses an emotion engine to generate deepfake video and audio data for the alarm based on the user's emotions.

[0500] Step 7:

[0501] The server transmits the generated video and audio data to the user terminal.

[0502] Step 8:

[0503] The device sounds an alarm and plays video and audio that matches the emotion.

[0504] Steps for processing song requests using emotion recognition

[0505] Step 1:

[0506] The user selects the song request function and the request screen is displayed.

[0507] Step 2:

[0508] The user inputs the title of the song they wish to request.

[0509] Step 3:

[0510] The terminal sends the request information to the server.

[0511] Step 4:

[0512] The server checks the copyright information of the requested song against its database.

[0513] Step 5:

[0514] If copyright permits, the server uses an emotion engine to generate deepfake video and audio data based on the user's emotions.

[0515] Step 6:

[0516] The server transmits the generated video and audio data to the user terminal.

[0517] Step 7:

[0518] The terminal plays back the received video and audio data, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[0519] Example 2

[0520] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0521] Although internet-based interactive communication technologies have become increasingly sophisticated in recent years, systems that recognize users' emotions in real time and generate personalized video and audio accordingly have yet to become widespread. Furthermore, to improve the user experience, there is a growing demand for systems that seamlessly integrate authentication processes, conversation functions, alarm settings, and song request functions in a way that responds to emotions. However, existing systems struggle to integrate multiple technologies and engines and provide interactions that respond to users' emotions. Furthermore, the processing efficiency and speed of data transmission and reception required to execute these functions in real time remain challenges.

[0522] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user authentication information, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting input data from the user into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for recognizing emotions from the user's input data, means for generating a response reflecting the recognized emotion, means for generating deepfake video and audio data based on the response, and means for transmitting the generated video and audio data to a user terminal and playing them. This enables personalized communication in real time according to the user's emotions. Furthermore, the seamless integration of multiple functions such as authentication, conversation, alarm setting, and song request can significantly improve the user experience.

[0523] "User authentication" is the process in which the server queries a database to verify the user's identity based on the authentication information entered by the user.

[0524] "Speech recognition" is a technology that analyzes voice data entered by a user and converts it into text data.

[0525] "Natural language processing" is a technology for analyzing input text data and generating appropriate responses.

[0526] "Emotion recognition" is a technology that extracts emotions from user input data and recognizes those emotions as data.

[0527] "Deepfake" is a technology that uses artificial intelligence to synthesize events and actions that do not actually exist into video data.

[0528] "Speech synthesis" is a technology that generates voice data based on generated text data.

[0529] A "user terminal" is a device that a user directly operates, and includes smartphones, tablets, PCs, etc.

[0530] A "database" is a system for storing user information, alarm settings, request information, and the like.

[0531] A "server" is a central computer device in a system that processes data and performs production processing.

[0532] "Authentication result" is data indicating whether user authentication was successful or unsuccessful as a result of the server querying the database.

[0533] A "home screen" is the initial interface that is displayed after a user is successfully authenticated.

[0534] An "error message" is a warning message that is displayed when user authentication fails.

[0535] "Response generation" is the process of constructing an appropriate reply to user input using natural language processing and emotion recognition.

[0536] "Video and audio data" refers to multimedia data generated using deepfake and voice synthesis technology.

[0537] "Real-time" refers to responding immediately to user input and executing processing without delay.

[0538] "Alarm settings" is a feature that allows users to set up messages and notifications that will be triggered at specific times.

[0539] A "song request" is the process by which a user requests that a particular song be played.

[0540] "Copyright Information" is data used to verify that the requested song is legally licensed.

[0541] MODE FOR CARRYING OUT THE INVENTION

[0542] System Configuration

[0543] This invention is a system for realizing two-way communication with users using a data processing device. This system utilizes multimodal generation AI with deepfake technology, a voice bot, and an emotion engine, enabling real-time interaction with users.

[0544] Basic components

[0545] The system consists of the following main components:

[0546] User devices: devices such as smartphones, tablets, and PCs

[0547] Server: A computer system responsible for data processing and generation.

[0548] Database: A storage system that stores user information, alarm settings, request information, etc.

[0549] AI engines: natural language processing engine, deepfake generation engine, speech synthesis engine, emotion engine

[0550] User Authentication

[0551] The user launches the app and enters their login information. The device encrypts the input information and sends it to the server, which then queries the database to obtain the authentication result. If authentication is successful, the home screen is displayed; if it is unsuccessful, an error message is displayed on the device. This uses data encryption using AES (Advanced Encryption Standard) and authentication using OAuth tokens.

[0552] Conversation function using emotion recognition

[0553] The user selects the conversation function and inputs a message by voice or text. The device recognizes the voice input and converts it into text. This process uses the Google Cloud Speech-to-Text API. The converted text data is sent to the server, where a natural language processing engine (e.g., OpenAI's GPT-3) generates an appropriate response. The server uses an emotion engine (e.g., IBM Watson's Tone Analyzer) to recognize the user's emotions and reflect them in the response. Finally, the server uses a deepfake generation engine (e.g., DeepFaceLab) and a speech synthesis engine (e.g., Amazon Polly) to generate video and audio, which are then sent to the device for playback.

[0554] Alarm setting using emotion recognition

[0555] The user opens the alarm settings screen and enters the desired alarm time and wake-up message. The device sends the setting information to the server, which stores the data in a database. When the alarm time comes, the server retrieves the corresponding alarm data and uses an emotion engine to generate deepfake video and audio that matches the user's emotions. The generated data is sent to the device, which then plays the video and audio along with the alarm. DeepFaceLab and Amazon Polly are used in this process.

[0556] Song Request Using Emotion Recognition

[0557] The user inputs a song request, and the device sends the request information to the server. The server then checks the database for copyright information. If the copyright is permitted, the server uses an emotion engine to recognize the user's emotions and generate deepfake video and audio. This uses DeepFaceLab and Amazon Polly. Finally, the generated data is sent to the device, where the user can enjoy the video and audio.

[0558] Specific examples

[0559] Example 1: Conversation function

[0560] When a user launches the app and logs in, the home screen appears, and they tap the conversation function icon to say, "Hello, how are you today?" The device converts the speech into text and sends it to the server. The server analyzes the text and generates a response such as, "I'm feeling good today, how about you?" The emotion engine recognizes emotions from the user's voice and reflects this in the response. The deepfake generation engine generates video that matches the user's emotions, and the speech synthesis engine adjusts the tone to generate the audio. The device plays back the video and audio, and the "oshi" character says, "I'm feeling good today, how about you?"

[0561] Example 2: Alarm settings

[0562] The user opens the alarm settings and sets it to "Good morning, it's time to wake up" at 7:00. When the alarm time comes, the server receives the message "Good morning, it's time to wake up" and uses the emotion engine to generate video and audio that matches the user's emotion. While the device sounds the alarm, it plays video and audio that matches the emotion, and the "oshi" wakes the user up.

[0563] Prompt Sentence Examples

[0564] "Simulate a scenario where a user uses emotion-aware conversation features."

[0565] "Please explain the steps a user goes through to set up an alarm using deepfake technology."

[0566] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0567] User Authentication

[0568] Step 1:

[0569] A user launches the app and accesses the login screen, where they enter their username and password.

[0570] Input: Username, Password

[0571] Output: User credentials

[0572] Step 2:

[0573] The terminal encrypts the entered username and password using AES (Advanced Encryption Standard) and sends an HTTP request.

[0574] Input: User credentials

[0575] Output: Encrypted user credentials

[0576] Step 3:

[0577] The server receives the encrypted user authentication information and queries the database to retrieve the corresponding user information.

[0578] Input: Encrypted user credentials

[0579] Output: Database query results

[0580] Step 4:

[0581] The server returns the authentication result (success or failure) in JSON format, including the home screen data for successful authentication.

[0582] Input: Database query results

[0583] Output: Authentication result (JSON format)

[0584] Step 5:

[0585] The device analyzes the authentication result it receives, and displays the home screen if authentication is successful, or an error message if authentication fails.

[0586] Input: Authentication result (JSON format)

[0587] Output: Home screen or error message

[0588] Conversation function using emotion recognition

[0589] Step 1:

[0590] The user selects the app's conversation feature and enters a message by voice or text.

[0591] Input: Voice or text message

[0592] Output: User message

[0593] Step 2:

[0594] The device converts voice input into text data using the Google Cloud Speech-to-Text API (when using voice input).

[0595] Input: Voice message

[0596] Output: Text data

[0597] Step 3:

[0598] The terminal converts the converted text data into JSON format and sends it to the server via the REST API.

[0599] Input: Text data

[0600] Output: JSON format text data

[0601] Step 4:

[0602] The server receives the text data and passes it to a natural language processing engine (e.g., OpenAI's GPT-3) for analysis.

[0603] Input: JSON format text data

[0604] Output: Response text

[0605] Step 5:

[0606] The natural language processing engine generates appropriate response text and returns it to the server.

[0607] Input: JSON format text data

[0608] Output: Response text

[0609] Step 6:

[0610] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize emotions from the user's text data.

[0611] Input: Response text

[0612] Output: Emotional information

[0613] Step 7:

[0614] Based on the emotional data recognized by the emotion engine, the appropriate emotion is reflected in the response text.

[0615] Input: Emotion information

[0616] Output: Sentiment-reflecting response text

[0617] Step 8:

[0618] The server generates video and audio using a deepfake generation engine (e.g., DeepFaceLab) and a speech synthesis engine (e.g., Amazon Polly).

[0619] Input: Sentiment-reflecting response text

[0620] Output: Video and audio data

[0621] Step 9:

[0622] The server encodes the generated video and audio data and sends it to the terminal as an HTTP response.

[0623] Input: Video and audio data

[0624] Output: Encoded video and audio data

[0625] Step 10:

[0626] The device decodes and plays the received video and audio data using a media player such as VLC Player.

[0627] Input: Encoded video and audio data

[0628] Output: Video and audio playback

[0629] Alarm setting using emotion recognition

[0630] Step 1:

[0631] The user opens the alarm setting screen and enters the desired alarm time and wake-up message.

[0632] Input: Alarm time and wake-up message

[0633] Output: Alarm setting information

[0634] Step 2:

[0635] The terminal converts the input alarm information into JSON format and sends it to the server.

[0636] Input: Alarm setting information

[0637] Output: Alarm configuration information in JSON format

[0638] Step 3:

[0639] The server stores the received alarm information in the database using an INSERT query.

[0640] Input: Alarm setting information in JSON format

[0641] Output: Database saved results

[0642] Step 4:

[0643] When the alarm time arrives, the server queries the corresponding alarm setting data.

[0644] Input: Database saved results (alarm time)

[0645] Output: Alarm setting data

[0646] Step 5:

[0647] The server uses an emotion engine to generate deepfake video and audio that matches the user's emotions.

[0648] Input: Alarm setting data

[0649] Output: Emotion-reflecting video and audio data

[0650] Step 6:

[0651] The server encodes the generated video and audio data and sends it to the terminal in an HTTP response.

[0652] Input: Emotion-reflecting video and audio data

[0653] Output: Encoded video and audio data

[0654] Step 7:

[0655] The terminal detects that the alarm time has come and plays the received video and audio data along with the alarm sound.

[0656] Input: Encoded video and audio data

[0657] Output: Alarm sound and video and audio playback

[0658] Song Request Using Emotion Recognition

[0659] Step 1:

[0660] The user opens the app's song request function and enters the song title and artist name.

[0661] Input: Song title and artist name

[0662] Output: Request information

[0663] Step 2:

[0664] The terminal converts the request information into JSON format and sends it to the server.

[0665] Input: Request information

[0666] Output: Request information in JSON format

[0667] Step 3:

[0668] The server queries the database for copyright information for the requested song, and if necessary consults external copyright databases via an API.

[0669] Input: Request information in JSON format

[0670] Output: Copyright information

[0671] Step 4:

[0672] If copyright permits, the server uses an emotion engine to recognize the user's emotions and generate deepfake video and audio, using DeepFaceLab and Amazon Polly.

[0673] Input: Copyright information

[0674] Output: Emotion-reflecting video and audio data

[0675] Step 5:

[0676] The server encodes the generated video and audio data and sends it to the terminal in an HTTP response.

[0677] Input: Emotion-reflecting video and audio data

[0678] Output: Encoded video and audio data

[0679] Step 6:

[0680] The device receives the data and plays the video and audio using a VLC player or similar, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[0681] Input: Encoded video and audio data

[0682] Output: Video and audio playback

[0683] (Application example 2)

[0684] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0685] Conventional data processing technologies have limited two-way communication with users, making it difficult to provide a personalized experience that reflects the user's emotions in real time. In particular, playback of video and audio content is not adjusted according to the user's emotional state, resulting in a lack of immersive and personalized entertainment. Therefore, there is a need to improve user satisfaction and provide a more immersive viewing experience.

[0686] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving authentication information entered by the user, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting input data from the user into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for generating deepfake video and audio data based on the response, means for transmitting the generated video and audio data to the user terminal and playing it, and means for adjusting the deepfake video and audio data based on the user's emotions using an emotion recognition engine that recognizes the user's emotions. This enables a personalized viewing experience according to the user's emotions.

[0687] The "data processing device" refers to the entire system for realizing two-way communication with the user.

[0688] "Authentication information" refers to information such as a username and password required for a user to access an application.

[0689] A "database" is a storage device for saving and managing data such as user information and request information.

[0690] The "home screen" is the main screen that is displayed when the user is successfully authenticated.

[0691] An "error message" is an error message that is displayed when a user fails authentication.

[0692] "Speech recognition" is the technique or process of analyzing and converting spoken input from a user into text data.

[0693] "Natural language processing" is a technology for analyzing text data from users and generating appropriate responses.

[0694] "Deepfake video" is a synthetic video generated using AI technology that has similar qualities to real-life video.

[0695] "Voice data" refers to data that includes a voice response to a user, and is generated by a voice synthesis engine.

[0696] An "emotion recognition engine" is a technology that analyzes a user's emotions from their facial expressions, voice, etc., and generates an appropriate response based on the results.

[0697] A "user terminal" is a device that is directly operated by a user, such as a smartphone, tablet, or PC.

[0698] This invention is a system that uses a data processing device to realize two-way communication with a user. Specifically, it utilizes an emotion recognition engine, a deepfake generation engine, and a voice synthesis engine to provide a personalized experience based on the user's emotions.

[0699] System Configuration

[0700] The system consists of the following main components:

[0701] 1. User Device:

[0702] These devices include smartphones, tablets, and PCs. These devices are operated by users and are equipped with cameras and microphones.

[0703] 2. Server:

[0704] Performs data processing and generation processes: receives data from user terminals and generates appropriate responses.

[0705] 3. Database:

[0706] Store and manage data such as user information, alarm settings, and request information.

[0707] 4. AI Engine:

[0708] It includes a natural language processing engine, a deepfake generation engine, a speech synthesis engine, and an emotion recognition engine.

[0709] Program Description

[0710] The whole system works in the following steps:

[0711] 1. User authentication:

[0712] The user accesses the application using their device and enters their authentication information. The device sends this information to the server, which then queries the database to obtain the authentication result. Depending on the result, the home screen or an error message is displayed.

[0713] 2. Emotion recognition:

[0714] When a user accesses the system while standing in front of the camera, the emotion recognition engine analyzes the user's facial expressions and voice to recognize their emotions.

[0715] 3. Deepfake Generation:

[0716] Based on the recognized emotional data, the natural language processing engine generates appropriate response text, and based on that, the deepfake generation engine and speech synthesis engine generate video and audio.

[0717] 4. Content playback:

[0718] The generated deepfake video and audio are then sent to the user's device and played back, allowing the user to enjoy a personalized viewing experience based on their emotions.

[0719] Specific examples

[0720] Example 1: Conversation function

[0721] The user launches the application and enters their login information for authentication. After successful authentication, the home screen is displayed.

[0722] The user selects the conversation function and types "How are you today?" The system converts this into text data through voice recognition and sends it to the server.

[0723] The server uses an emotion recognition engine to analyze the user's emotions, and a natural language processing engine generates a response such as "I'm feeling good today, how about you?"

[0724] Based on the emotional data of the response, the deepfake generation engine and voice synthesis engine generate video and audio, which are then sent to the user's device.

[0725] The user terminal plays the generated video and audio, providing the user with a real-time interactive communication experience.

[0726] Example 2: Alarm settings

[0727] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[0728] When the alarm time arrives, the server retrieves the relevant data and uses an emotion recognition engine to generate deepfake video and audio that matches the user's emotions.

[0729] The user terminal sounds an alarm with the generated video and audio, waking the user up in an emotionally appropriate way.

[0730] Prompt Sentence Examples

[0731] "Please provide step-by-step instructions showing how to play personalized video and audio that feels natural to the user. Include instructions for adjusting video and audio presentation based on the user's emotions."

[0732] In this way, the system can provide a personalized viewing experience according to the user's emotions.

[0733] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0734] Step 1:

[0735] A user starts an application and enters login information (username, password). The terminal receives this login information and sends it to the server to query the database. Based on this input, the server looks up the authentication database and returns the result (authentication success or failure).

[0736] Step 2:

[0737] The server queries the database and sends information on whether authentication was successful or not to the device. If authentication was successful, the home screen is displayed, and if it was not, an error message is displayed. This display guides the user on what to do next.

[0738] Step 3:

[0739] The user selects the conversation function from the home screen and inputs either voice or text. In the case of voice input, the device uses a voice recognition API to convert the voice data into text data and sends it to the server. The server receives this text data and analyzes it.

[0740] Step 4:

[0741] The server sends the received text data to the natural language processing engine and begins analysis. The natural language processing engine generates an appropriate response text based on the user's input. This analysis and response generation is performed to output an accurate response to the user's question or request.

[0742] Step 5:

[0743] The server sends the generated response text and data for recognizing the user's emotions to the emotion recognition engine. The emotion recognition engine recognizes the user's emotions (e.g., joy, anger, sadness, and happiness) from the input audio or video data and outputs the results.

[0744] Step 6:

[0745] Based on the emotion recognition results, the server sends data to a deepfake generation engine and a speech synthesis engine to generate response video and audio. During this generation process, the facial expression of the video and the tone of the voice are adjusted based on the emotion information. This procedure provides the user with a response that is emotionally relevant.

[0746] Step 7:

[0747] The server then transmits the generated video and audio data to the user's device, where it is played back. The user receives personalized responses in real time using both video and audio, creating a two-way communication experience. This playback process allows the user to enjoy enhanced visual and audio interactions.

[0748] Based on these steps, the present invention realizes a technique for recognizing a user's emotions in real time and generating and providing appropriate responses based thereon.

[0749] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0750] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0751] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0752] [Second embodiment]

[0753] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0754] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0755] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0756] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0757] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0758] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0759] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0760] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0761] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0762] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0763] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0764] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0765] This invention is a system that allows users to enjoy interactive communication with their "oshi" using a data processing device. Specifically, it utilizes multimodal generation AI with deepfake technology and voice bots, while taking privacy and copyright protection into consideration.

[0766] System Configuration

[0767] This system consists of a user terminal, a server, a database, and an AI engine (natural language processing engine, deepfake generation engine, and voice synthesis engine). The specific operation of each element is explained below.

[0768] User Authentication

[0769] 1. The user launches the app and enters their login information.

[0770] 2. The device sends the login information to the server.

[0771] 3. The server queries the database for login information and obtains the authentication result.

[0772] 4. The server sends the user's home screen if authentication is successful, and an error message if authentication fails.

[0773] 5. The device displays the home screen or an error message.

[0774] Conversation feature

[0775] 1. The user selects the conversation feature and enters information by voice or text.

[0776] 2. The device converts voice input into text (when using voice input).

[0777] 3. The terminal sends the entered text data to the server.

[0778] 4. The server analyzes the text data using a natural language processing engine and generates an appropriate response.

[0779] 5. The server generates response video and audio using a deepfake generation engine and a voice synthesis engine.

[0780] 6. The server transmits the generated video and audio data to the user terminal.

[0781] 7. The device plays video and audio, allowing the user to have a conversation with their favorite idol.

[0782] Alarm settings

[0783] 1. The user sets the alarm time and message.

[0784] 2. The device sends the configuration information to the server.

[0785] 3. The server stores the configuration information in a database.

[0786] 4. When the alarm time arrives, the server generates deepfake video and audio and sends them to the user's device.

[0787] 5. The device will play an alarm along with video and audio, and your favorite idol will wake you up.

[0788] Song Requests

[0789] 1. The user enters song request information.

[0790] 2. The device sends the request information to the server.

[0791] 3. The server checks the copyright information for the requested song.

[0792] 4. If copyright is permitted, the server generates deepfake video and audio and sends them to the user's device.

[0793] 5. The device plays the video and audio, and the user enjoys watching their favorite idol sing the song.

[0794] Specific examples

[0795] Example 1: Conversation function

[0796] When a user launches the app and logs in, the home screen appears.

[0797] The user taps the conversation icon and says, "Hello, how are you today?"

[0798] The device converts the speech into text and sends it to the server.

[0799] The server generates a response saying, "How are you today? How are you?", creating deepfake video and audio.

[0800] When the device plays the video and audio, your favorite idol will say, "I'm feeling good today. How about you?"

[0801] Example 2: Alarm settings

[0802] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[0803] When the alarm time comes, the server generates video and audio saying "Good morning, it's time to wake up" and sends it to the device.

[0804] The device will sound an alarm while playing video and audio, and the "oshi" will wake the user up.

[0805] The system of the present invention allows users to enjoy two-way communication with their "oshi" (favorite idols). It also thoroughly protects privacy and copyrights, promoting sound use.

[0806] The processing flow will be explained below.

[0807] Conversation function processing steps

[0808] Step 1:

[0809] The user launches the application and the login screen appears.

[0810] Step 2:

[0811] The user enters login information (username, password) and taps the login button.

[0812] Step 3:

[0813] The terminal sends the entered login information to the server.

[0814] Step 4:

[0815] The server queries the database for login information and authenticates the user.

[0816] Step 5:

[0817] The server returns the authentication result to the device. If authentication is successful, it returns the home screen data, and if it is unsuccessful, it returns an error message.

[0818] Step 6:

[0819] The device displays the home screen or an error message.

[0820] Step 7:

[0821] The user taps the conversation feature icon on the home screen and enters a message by voice or text.

[0822] Step 8:

[0823] The device calls the speech recognition API to convert voice input into text (when voice input is performed).

[0824] Step 9:

[0825] The terminal transmits the text data to the server.

[0826] Step 10:

[0827] The server passes the text data to a natural language processing engine and begins analysis.

[0828] Step 11:

[0829] A natural language processing engine generates appropriate response text.

[0830] Step 12:

[0831] The server passes the response text to the deepfake generation engine and speech synthesis engine.

[0832] Step 13:

[0833] The deepfake generation engine generates the video response, and the speech synthesis engine generates the audio response.

[0834] Step 14:

[0835] The server transmits the generated video and audio data to the terminal.

[0836] Step 15:

[0837] The device plays back the received video and audio data, enabling conversation with the "favorite."

[0838] Alarm configuration process steps

[0839] Step 1:

[0840] The user selects the alarm setting function and displays the setting screen.

[0841] Step 2:

[0842] The user inputs the alarm time and the message they want to wake up with.

[0843] Step 3:

[0844] The terminal sends the set alarm time and message to the server.

[0845] Step 4:

[0846] The server stores the alarm configuration information in a database.

[0847] Step 5:

[0848] When the alarm time arrives, the server retrieves the saved setting data.

[0849] Step 6:

[0850] The server generates deepfake video and audio data for the alarm.

[0851] Step 7:

[0852] The server transmits the generated data to the terminal.

[0853] Step 8:

[0854] The device will sound an alarm and play deepfake video and audio.

[0855] Song request processing steps

[0856] Step 1:

[0857] The user selects the song request function and the request screen is displayed.

[0858] Step 2:

[0859] The user inputs the title of the song they wish to request.

[0860] Step 3:

[0861] The terminal sends the request information to the server.

[0862] Step 4:

[0863] The server checks the copyright information of the requested song against its database.

[0864] Step 5:

[0865] If copyright permits, the server generates the deepfake video and audio data.

[0866] Step 6:

[0867] The server transmits the generated data to the terminal.

[0868] Step 7:

[0869] The device plays back the received video and audio data and provides a video of the "favorite" person singing the song.

[0870] Example 1

[0871] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0872] With current information processing technology, it is difficult for users to enjoy interactive communication with virtual characters in real time. Furthermore, there is a lack of systems that can generate high-quality audio and video while fully considering privacy and copyright protection. This limits the user experience and reduces satisfaction. Furthermore, implementing features such as alarms and song requests is complex, creating a demand for more user-friendly systems.

[0873] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0874] In this invention, the server includes means for receiving authentication information entered by a user, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting the user's input data into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for generating deepfake video and audio data based on the response, means for transmitting the generated video and audio data to a user terminal and playing it back, means for encrypting the data using SSL / TLS, means for understanding the context and generating a response using a natural language processing engine, and means for generating deepfake technology in high resolution and in real time. This allows users to enjoy high-quality two-way communication in real time through audio and video, and also allows them to use the system safely while protecting their privacy and copyright.

[0875] "User" refers to an individual or organization that uses the system.

[0876] "Authentication information" is information for verifying the identity of a user, and includes a username and password.

[0877] A "database" is a system for efficiently storing and managing information.

[0878] "Home screen" refers to the main operation screen that is displayed after a user has successfully logged in.

[0879] An "error message" is a warning or explanatory message that is sent to the user when an error occurs in the system.

[0880] "Speech recognition" is a technology that converts input speech into text data.

[0881] "Text data" is data that is stored or processed as textual information.

[0882] "Natural language processing" is a technology that allows computers to understand and analyze human language.

[0883] "Deepfake" is a technology that uses artificial intelligence to generate or alter video and audio.

[0884] "Audio data" is data that is stored or processed as audio information.

[0885] "Transmitting" is the act of transferring data from one device to another.

[0886] "Playback" refers to the act of outputting stored audio or video in a form that can be confirmed visually or audibly.

[0887] "Encryption" is a technology that converts data according to rules to make it unreadable to third parties.

[0888] An "alarm" is a function that notifies or warns the user at a specific time set by the user.

[0889] "Copyright" refers to the legal right to protect creative works.

[0890] "Request" means a request from a User to take a particular action or provide data.

[0891] The present invention is a system that allows a user to enjoy two-way communication with a virtual character using a data processing device. Specific embodiments of the present invention will be described below.

[0892] System Configuration

[0893] This system consists of a user terminal, a server, a database, and an AI engine (natural language processing engine, deepfake generation engine, and voice synthesis engine).

[0894] User Authentication

[0895] A user launches the app and enters their login information. The device sends the entered login information to the server. The server queries the database for the login information and obtains the authentication result. If authentication is successful, the server generates a home screen and sends it to the device. If authentication fails, the server generates an error message and sends it to the device. The device displays the home screen or the error message.

[0896] Conversation feature

[0897] The user selects the conversation function and inputs voice or text. The device converts the voice input into text (in the case of voice input). The device sends the converted text data to the server. The server analyzes it using a natural language processing engine and generates an appropriate response. The server creates a response video and audio using a deepfake generation engine and a speech synthesis engine. The server sends the generated video and audio data to the user's device. The device plays the video and audio, and the user enjoys a conversation with their "favorite character."

[0898] Alarm settings

[0899] The user opens the alarm settings and enters the alarm time and message to set it. The device sends the setting information to the server. The server saves the setting information in a database. When the alarm time arrives, the server generates deepfake video and audio and sends it to the user's device. The device plays the video and audio along with the alarm time, and the "oshi" wakes the user up.

[0900] Song Requests

[0901] The user inputs song request information on the app. The device sends the request information to the server. The server checks the copyright information of the requested song in a database. If the copyright is permitted, the server generates deepfake video and audio and sends it to the user's device. The device plays the video and audio, allowing the user to enjoy watching their favorite idol sing.

[0902] Hardware and software used

[0903] User devices: smartphones, tablets, PCs

[0904] Server: High-performance computer, cloud server

[0905] Database: SQL database, NoSQL database

[0906] AI engines: natural language processing engines (e.g., BERT), deepfake generation engines, speech synthesis engines (e.g., Tacotron)

[0907] Specific examples

[0908] Example 1: Conversation function

[0909] When a user launches the app and logs in, the home screen appears.

[0910] The user taps the conversation icon and says, "Hello, how are you today?"

[0911] The device converts the speech into text and sends it to the server.

[0912] The server generates a response saying, "How are you today? How are you?", creating deepfake video and audio.

[0913] When the device plays the video and audio, your favorite idol will say, "I'm feeling good today. How about you?"

[0914] Example 2: Alarm settings

[0915] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[0916] When the alarm time comes, the server generates video and audio saying "Good morning, it's time to wake up" and sends it to the device.

[0917] The device will sound an alarm while playing video and audio, and the "oshi" will wake the user up.

[0918] Prompt Sentence Examples

[0919] 1. "What happens when a user selects the conversation feature and types, 'Hello, how are you today?'"

[0920] 2. "What happens when a user sets an alarm for 7:00 and is woken up by a specified message?"

[0921] The system of the present invention allows users to enjoy two-way communication with their "oshi" and ensures thorough privacy and copyright protection.

[0922] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0923] User authentication process steps

[0924] Step 1:

[0925] The user launches the app and enters their username and password on the login screen.

[0926] Input: The username and password entered by the user

[0927] Output: Login information prepared to be sent to the terminal

[0928] Specific behavior: A login form is displayed on the user interface.

[0929] Step 2:

[0930] The terminal sends the entered username and password to the server.

[0931] Input: Login information (username and password)

[0932] Output: Login information sent to the server

[0933] Specific operation: Login information is encrypted with SSL / TLS and sent to the server.

[0934] Step 3:

[0935] The server queries the database for the received login information and searches for corresponding user data.

[0936] Input: Login information sent to the server

[0937] Output: Authentication result from the database

[0938] What happens: A search query is executed against the database using the username and password.

[0939] Step 4:

[0940] The server determines whether the authentication was successful or not based on the results obtained from the database.

[0941] Input: Authentication results from the database

[0942] Output: Home screen data or error message

[0943] Specific operation: If authentication is successful, home screen data based on the user information is generated; if authentication fails, an error message is generated.

[0944] Step 5:

[0945] The terminal displays the home screen data or an error message received from the server.

[0946] Input: Home screen data or error message sent from the server

[0947] Output: The home screen or error message that the user sees

[0948] Specific Behavior: The display screen is updated and an appropriate message is displayed to the user.

[0949] Conversation function processing steps

[0950] Step 1:

[0951] The user selects the conversation function and enters information by voice or text.

[0952] Input: Voice or text input from the user

[0953] Output: Input data received on the terminal

[0954] Specific behavior: The conversation icon is tapped, and an interface for voice or text input appears.

[0955] Step 2:

[0956] The device converts the input voice into text.

[0957] Input: Audio data (in the case of audio input)

[0958] Output: Text data

[0959] Specific operation: The voice recognition engine analyzes the voice data and converts it into text data.

[0960] Step 3:

[0961] The terminal transmits the converted text data to the server.

[0962] Input: Text data (including voice input)

[0963] Output: Text data sent to the server

[0964] Specific operation: The text data is encrypted and sent to the server.

[0965] Step 4:

[0966] The server analyzes the received data using a natural language processing engine and generates an appropriate response.

[0967] Input: Text data sent to the server

[0968] Output: Response text data

[0969] What it does: A natural language processing engine analyzes text data and generates contextual responses.

[0970] Step 5:

[0971] Based on the response generated by the server, video and audio are generated using a deepfake generation engine and a voice synthesis engine.

[0972] Input: Response text data

[0973] Output: Response video and audio data

[0974] Specific operation: The deepfake generation engine generates a response video, and the speech synthesis engine generates a response voice.

[0975] Step 6:

[0976] The server transmits the generated video and audio data to the user terminal.

[0977] Input: Response video and audio data

[0978] Output: Video and audio data sent to the user's device

[0979] What it does: Data is encrypted and efficiently transmitted to the device.

[0980] Step 7:

[0981] The device plays back the video and audio received.

[0982] Input: Video and audio data sent from the server

[0983] Output: The video and audio played to the user

[0984] Specific operation: Video and audio are played, and the user enjoys a conversation with their "favorite."

[0985] Alarm configuration process steps

[0986] Step 1:

[0987] The user opens the alarm settings and sets the alarm time and message.

[0988] Input: Alarm time and message

[0989] Output: Configuration information received on the device

[0990] Specific operation: The alarm setting screen is displayed and the user enters the time and message.

[0991] Step 2:

[0992] The terminal sends the setting information to the server.

[0993] Input: Alarm time and message

[0994] Output: Configuration information sent to the server

[0995] Specific operation: The configuration information is encrypted and sent to the server.

[0996] Step 3:

[0997] The server stores the configuration information in a database.

[0998] Input: Configuration information sent to the server

[0999] Output: Configuration information stored in the database

[1000] Specific behavior: High availability settings are made when data is saved to the database.

[1001] Step 4:

[1002] When the alarm time arrives, the server generates deepfake video and audio and sends them to the user's device.

[1003] Input: Saved alarm time and message

[1004] Output: Generated deepfake video and audio data

[1005] Specific operation: The alarm is triggered at the alarm time, and video and audio are generated and transmitted.

[1006] Step 5:

[1007] When the alarm time comes on the device, it plays video and audio.

[1008] Input: Video and audio data sent from the server

[1009] Output: The video and audio played to the user

[1010] Specific operation: Video and audio are played in sync with the alarm sound, and the "oshi" character wakes up the user.

[1011] Song request processing steps

[1012] Step 1:

[1013] The user enters song request information on the app.

[1014] Input: Song request information

[1015] Output: Request information received on the device

[1016] Specific operation: A request screen is displayed, and the user enters the desired song title and artist name.

[1017] Step 2:

[1018] The terminal sends the request information to the server.

[1019] Input: Request information

[1020] Output: Request information sent to the server

[1021] Specific operation: The request information is encrypted and sent to the server.

[1022] Step 3:

[1023] The server checks the copyright information of the requested song against its database.

[1024] Input: Request information sent to the server

[1025] Output: Copyright information check result

[1026] What it does: Query the database for copyright information for the requested song.

[1027] Step 4:

[1028] If copyright is permitted, the server generates deepfake video and audio and sends them to the user's device.

[1029] Input: Copyright permission confirmation result, requested song information

[1030] Output: Generated deepfake video and audio data

[1031] Specific operation: The generated video and audio are encrypted and sent to the user terminal.

[1032] Step 5:

[1033] The device plays the video and audio, and the user enjoys watching their favorite idol sing.

[1034] Input: Video and audio data sent from the server

[1035] Output: The video and audio played to the user

[1036] Specific operation: The playback screen is displayed and video and audio are played.

[1037] This concludes the detailed explanation of each processing step in this system, which allows users to enjoy high-quality two-way communication in real time.

[1038] (Application example 1)

[1039] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1040] Conventional advertising systems have difficulty providing interactive and effective advertisements tailored to users' interests. There is a particular demand for advertising that utilizes "favorites" that users feel attached to, but current technology does not provide an effective means for achieving this. Furthermore, if real-time responses to user input are required, advanced technology is required to achieve this. An effective system is needed to solve these problems.

[1041] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1042] In this invention, the server includes a means for a user to select an advertisement display, a means for generating advertising video and audio data featuring the user's "oshi" using a deepfake generation engine and a voice bot, a means for delivering and playing the generated advertising video and audio data to the user's terminal, a means for receiving questions and comments from the user, analyzing them with natural language processing means, and generating appropriate responses, and a means for playing the generated responses via the user's "oshi" using the natural language processing means and the voice bot. This allows users to receive responses in real time while enjoying interactive advertising via their "oshi."

[1043] The "means for receiving authentication information entered by a user" refers to a component or system for receiving login information and authentication information entered by a user via a network.

[1044] The "means for querying a database for authentication information and obtaining an authentication result" refers to a process or system for querying a database for received authentication information, verifying its validity, and obtaining an authentication result.

[1045] "Means for displaying the user's home screen or an error message based on the authentication result" refers to an interface or software for displaying an appropriate screen (home screen or an error message) on the user's device depending on the authentication result.

[1046] "Means for recognizing voice input data from a user and converting it into text data" refers to technology or software that converts voice input data into text format.

[1047] The "natural language processing means for analyzing text data and generating an appropriate response" refers to a natural language processing engine or algorithm for analyzing text data and generating an appropriate response.

[1048] A "means for generating deepfake video and audio data based on responses" is a process or system that uses deepfake technology to create video and audio data based on generated responses.

[1049] "Means for transmitting the generated video and audio data to a user terminal and playing it" refers to a device or application for transmitting the generated deepfake video and audio data to a user's device via a network and playing it.

[1050] The "means by which a user can select to display an advertisement" refers to an interface or software that allows a user to select to view an advertisement.

[1051] "Means for generating advertising video and audio data featuring a user's favorite character using a deepfake generation engine and voice bot" refers to an engine or system that makes full use of deepfake technology and voice bots to create advertising video and audio featuring a user's favorite character or idol.

[1052] "Means for delivering and playing the generated advertising video and audio data to a user terminal" refers to a method or apparatus for transmitting the generated advertising video and audio data to a user's device via a network and automatically playing them.

[1053] "Means for receiving questions and comments from users, analyzing them using natural language processing means, and generating appropriate responses" refers to natural language processing technology for receiving questions and comments from users, analyzing them, and creating appropriate responses.

[1054] "Means for using the natural language processing means and a voice bot to play back the generated response through the user's "oshi"" refers to a system or method for using a voice bot to play back a response generated by natural language processing in the voice of the user's "oshi."

[1055] This invention is an interactive advertising display system that uses the user's "oshi" (favorites) and utilizes deepfake technology and voice bots to provide advertisements featuring people special to the user. The following describes how this system can be specifically realized.

[1056] System Configuration

[1057] The system consists of the following main components:

[1058] 1. User device: Mobile device such as smartphone (iOS / Android)

[1059] 2. Server: A cloud-based server (e.g., an AWS EC2 instance)

[1060] 3. Database: A database for data storage and authentication (e.g., MySQL on AWS RDS)

[1061] 4. AI engines: Deepfake generation engines, voice bots, natural language processing engines (e.g., Google Cloud NLP API, TensorFlow, Twilio Voice API)

[1062] User Authentication

[1063] 1. The user launches the app and enters their login information.

[1064] 2. The device sends the login information to the cloud server.

[1065] 3. The server queries the database for login information and obtains the authentication result.

[1066] 4. If the server is successful, it sends the user's home screen; if not, it sends an error message.

[1067] 5. The device displays the home screen or an error message.

[1068] Ad display function

[1069] 1. The user clicks (or taps) the button to view the ad.

[1070] 2. The server uses a deepfake generation engine and a voice bot to generate advertising video and audio data featuring the user's "favorite person."

[1071] 3. The server distributes the generated video and audio data to the user terminal.

[1072] 4. The device plays the advertising video and audio, and the "oshi" delivers the advertisement to the user.

[1073] Interactive Features

[1074] 1. Users can type in questions or comments during the ad.

[1075] 2. The terminal sends the user's input to the server.

[1076] 3. The server analyzes the user's input using a natural language processing engine and generates an appropriate response.

[1077] 4. The server uses deepfakes and a voice bot to generate video and audio data of the "favorite" responding.

[1078] 5. The terminal plays a video and audio response to the user.

[1079] Recommendations

[1080] 1. The server analyzes the user's past behavior data in the database.

[1081] 2. The server selects the most suitable advertisement based on the analysis results.

[1082] 3. The server generates the selected advertisement using a deepfake generation engine and a voice bot and delivers it to the user's device.

[1083] Specific prompt examples

[1084] A prompt that the user types in the app: "How do I use this product?"

[1085] Prompt the server-side AI model: "Please explain in the best possible terms how to use the product the user cares about."

[1086] In this way, the system of the present invention provides a technique that allows for real-time responses while providing interactive advertising via the user's "oshi."

[1087] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1088] Step 1:

[1089] A user launches the app and enters their login information, which includes their user ID and password, and is sent to the device.

[1090] Step 2:

[1091] The terminal sends the received login information to the cloud server. Specifically, the login information is sent to the cloud server using the HTTPS protocol.

[1092] Step 3:

[1093] The server receives the login information and queries a database, which stores the user's authentication information, to verify its validity.

[1094] Step 4:

[1095] The server obtains the authentication result, generates the user's home screen data if successful, or generates an error message if unsuccessful, and sends this information to the device.

[1096] Step 5:

[1097] The device displays the home screen or an error message, using an HTML rendering engine to display the appropriate screen.

[1098] Step 6:

[1099] The user selects to display an ad. A button click is registered as input, and the device sends this information to the server.

[1100] Step 7:

[1101] The server receives the button click information and launches the deepfake generation engine and voice bot. The deepfake generation engine generates video using TensorFlow, and the voice bot generates audio using the Twilio API.

[1102] Step 8:

[1103] The server transmits the generated advertising video and audio data to the user terminal, where the data is transmitted in compressed form and decoded.

[1104] Step 9:

[1105] The device plays the advertisement video and audio, using the device's video and audio players.

[1106] Step 10:

[1107] The user enters questions or comments during the advertisement. The input is text data, which is sent from the terminal to the server.

[1108] Step 11:

[1109] The server receives the user's input and analyzes it using a natural language processing engine (Google Cloud NLP API), where the input text is parsed and the appropriate response data is generated.

[1110] Step 12:

[1111] The server processes the response data using a deepfake generation engine and voice bot, generating video and audio data in which the "favorite" responds.

[1112] Step 13:

[1113] The server transmits the generated response video and audio data to the user terminal, where the data is recompressed and transmitted.

[1114] Step 14:

[1115] The device plays back the response video and audio, allowing the user to receive a response from their "oshi" in real time.

[1116] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1117] System Configuration

[1118] This invention is a system for realizing two-way communication with users using a data processing device. Specifically, it utilizes deepfake technology, a voice bot, and a multimodal generation AI using an emotion engine to realize real-time interaction with users.

[1119] Basic components

[1120] The system consists of the following main components:

[1121] User devices: smartphones, tablets, PCs, etc.

[1122] Server: Data processing and generation processing

[1123] Database: Stores user information, alarm settings, request information, etc.

[1124] AI engines: natural language processing engine, deepfake generation engine, speech synthesis engine, emotion engine

[1125] User Authentication

[1126] 1. The user launches the app and enters their login information (username, password).

[1127] 2. The device sends the input information to the server.

[1128] 3. The server queries the database for login information and obtains the authentication result.

[1129] 4. The server returns the authentication result to the device, and if authentication is successful, the home screen is displayed, and if it is unsuccessful, an error message is displayed.

[1130] 5. The device displays the home screen or an error message.

[1131] Conversation function using emotion recognition

[1132] 1. The user selects the conversation feature and enters information by voice or text.

[1133] 2. The device uses a speech recognition API to convert voice input into text (when voice input is used).

[1134] 3. The terminal sends the entered text data to the server.

[1135] 4. The server passes the text data to the natural language processing engine and begins analysis.

[1136] 5. The natural language processing engine generates the appropriate response text.

[1137] 6. The server uses the emotion engine to recognize emotions from the user's input data.

[1138] 7. The emotional information recognized by the emotion engine is reflected in response generation.

[1139] 8. The server generates a response video and audio using a deepfake generation engine and a voice synthesis engine, adjusting the facial expression and tone of the audio based on the recognized user emotion.

[1140] 9. The server sends the generated video and audio data to the terminal.

[1141] 10. The device plays video and audio, allowing the user to have a conversation with their favorite idol.

[1142] Alarm setting using emotion recognition

[1143] 1. The user sets the alarm time and message.

[1144] 2. The device sends the configuration information to the server.

[1145] 3. The server stores the configuration information in a database.

[1146] 4. When the alarm time arrives, the server retrieves the data and uses an emotion engine to generate deepfake video and audio data that matches the user's emotions.

[1147] 5. The server sends the generated data to the device.

[1148] 6. The device sounds an alarm and plays emotionally appropriate video and audio to wake the user up.

[1149] Song Request Using Emotion Recognition

[1150] 1. The user enters song request information.

[1151] 2. The device sends the request information to the server.

[1152] 3. The server checks the copyright information of the requested song against its database.

[1153] 4. If copyright permits, the server uses an emotion engine to generate deepfake video and audio data based on the user's emotions.

[1154] 5. The server sends the generated data to the device.

[1155] 6. The device plays the video and audio data, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[1156] Specific examples

[1157] Example 1: Conversation function

[1158] When a user launches the app and logs in, the home screen appears.

[1159] The user taps the conversation icon and says, "Hello, how are you today?"

[1160] The device converts the speech into text and sends it to the server.

[1161] The server performs text analysis and generates a response: "I'm fine today, how about you?"

[1162] The emotion engine recognizes emotions from the user's voice and reflects them in the response.

[1163] The deepfake generation engine generates images that match the user's emotions, and the voice synthesis engine adjusts the tone to generate voice.

[1164] The device plays video and audio, and your favorite idol speaks to you, saying, "I'm feeling good today. How about you?"

[1165] Example 2: Alarm settings

[1166] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[1167] When the alarm time comes, the server receives the message "Good morning, it's time to wake up" and uses an emotion engine to generate video and audio that matches the user's emotions.

[1168] The device sounds an alarm and plays video and audio that matches the user's emotions, and the ``oshi'' character wakes the user up.

[1169] The system of the present invention allows users to enjoy real-time interactive communication with their "oshi" and provides a personalized experience based on their emotions. It also thoroughly protects privacy and copyrights and promotes the healthy use of technology.

[1170] The processing flow will be explained below.

[1171] Processing steps for conversational functions using emotion recognition

[1172] Step 1:

[1173] The user launches the application and the login screen appears.

[1174] Step 2:

[1175] The user enters login information (username, password) and taps the login button.

[1176] Step 3:

[1177] The terminal sends the entered login information to the server.

[1178] Step 4:

[1179] The server queries the database for login information and authenticates the user.

[1180] Step 5:

[1181] The server returns the authentication result to the device. If authentication is successful, it returns the home screen data, and if it is unsuccessful, it returns an error message.

[1182] Step 6:

[1183] The device displays the home screen or an error message.

[1184] Step 7:

[1185] The user taps the conversation feature icon and enters a message by voice or text.

[1186] Step 8:

[1187] The device calls the speech recognition API to convert voice input into text (when voice input is performed).

[1188] Step 9:

[1189] The terminal transmits the text data to the server.

[1190] Step 10:

[1191] The server passes the text data to a natural language processing engine and begins analysis.

[1192] Step 11:

[1193] A natural language processing engine generates appropriate response text.

[1194] Step 12:

[1195] The server uses an emotion engine to recognize emotions from the user's input data.

[1196] Step 13:

[1197] The emotional information recognized by the emotion engine is reflected in response generation.

[1198] Step 14:

[1199] The server uses a deepfake generation engine and a voice synthesis engine to generate a response video and audio, adjusting the facial expression and tone of the audio based on the user's recognized emotions.

[1200] Step 15:

[1201] The server transmits the generated video and audio data to the terminal.

[1202] Step 16:

[1203] The device plays back the received video and audio data, enabling conversation with the "favorite."

[1204] Processing steps for setting alarms using emotion recognition

[1205] Step 1:

[1206] The user selects the alarm setting function and displays the setting screen.

[1207] Step 2:

[1208] The user inputs the alarm time and the message they want to wake up with.

[1209] Step 3:

[1210] The terminal sends the set alarm time and message to the server.

[1211] Step 4:

[1212] The server stores the alarm configuration information in a database.

[1213] Step 5:

[1214] When the alarm time arrives, the server acquires the set alarm information.

[1215] Step 6:

[1216] The server uses an emotion engine to generate deepfake video and audio data for the alarm based on the user's emotions.

[1217] Step 7:

[1218] The server transmits the generated video and audio data to the user terminal.

[1219] Step 8:

[1220] The device sounds an alarm and plays video and audio that matches the emotion.

[1221] Steps for processing song requests using emotion recognition

[1222] Step 1:

[1223] The user selects the song request function and the request screen is displayed.

[1224] Step 2:

[1225] The user inputs the title of the song they wish to request.

[1226] Step 3:

[1227] The terminal sends the request information to the server.

[1228] Step 4:

[1229] The server checks the copyright information of the requested song against its database.

[1230] Step 5:

[1231] If copyright permits, the server uses an emotion engine to generate deepfake video and audio data based on the user's emotions.

[1232] Step 6:

[1233] The server transmits the generated video and audio data to the user terminal.

[1234] Step 7:

[1235] The terminal plays back the received video and audio data, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[1236] Example 2

[1237] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1238] Although internet-based interactive communication technologies have become increasingly sophisticated in recent years, systems that recognize users' emotions in real time and generate personalized video and audio accordingly have yet to become widespread. Furthermore, to improve the user experience, there is a growing demand for systems that seamlessly integrate authentication processes, conversation functions, alarm settings, and song request functions in a way that responds to emotions. However, existing systems struggle to integrate multiple technologies and engines and provide interactions that respond to users' emotions. Furthermore, the processing efficiency and speed of data transmission and reception required to execute these functions in real time remain challenges.

[1239] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user authentication information, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting input data from the user into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for recognizing emotions from the user's input data, means for generating a response reflecting the recognized emotion, means for generating deepfake video and audio data based on the response, and means for transmitting the generated video and audio data to a user terminal and playing them. This enables personalized communication in real time according to the user's emotions. Furthermore, the seamless integration of multiple functions such as authentication, conversation, alarm setting, and song request can significantly improve the user experience.

[1240] "User authentication" is the process in which the server queries a database to verify the user's identity based on the authentication information entered by the user.

[1241] "Speech recognition" is a technology that analyzes voice data entered by a user and converts it into text data.

[1242] "Natural language processing" is a technology for analyzing input text data and generating appropriate responses.

[1243] "Emotion recognition" is a technology that extracts emotions from user input data and recognizes those emotions as data.

[1244] "Deepfake" is a technology that uses artificial intelligence to synthesize events and actions that do not actually exist into video data.

[1245] "Speech synthesis" is a technology that generates voice data based on generated text data.

[1246] A "user terminal" is a device that a user directly operates, and includes smartphones, tablets, PCs, etc.

[1247] A "database" is a system for storing user information, alarm settings, request information, and the like.

[1248] A "server" is a central computer device in a system that processes data and performs production processing.

[1249] "Authentication result" is data indicating whether user authentication was successful or unsuccessful as a result of the server querying the database.

[1250] A "home screen" is the initial interface that is displayed after a user is successfully authenticated.

[1251] An "error message" is a warning message that is displayed when user authentication fails.

[1252] "Response generation" is the process of constructing an appropriate reply to user input using natural language processing and emotion recognition.

[1253] "Video and audio data" refers to multimedia data generated using deepfake and voice synthesis technology.

[1254] "Real-time" refers to responding immediately to user input and executing processing without delay.

[1255] "Alarm settings" is a feature that allows users to set up messages and notifications that will be triggered at specific times.

[1256] A "song request" is the process by which a user requests that a particular song be played.

[1257] "Copyright Information" is data used to verify that the requested song is legally licensed.

[1258] MODE FOR CARRYING OUT THE INVENTION

[1259] System Configuration

[1260] This invention is a system for realizing two-way communication with users using a data processing device. This system utilizes multimodal generation AI with deepfake technology, a voice bot, and an emotion engine, enabling real-time interaction with users.

[1261] Basic components

[1262] The system consists of the following main components:

[1263] User devices: devices such as smartphones, tablets, and PCs

[1264] Server: A computer system responsible for data processing and generation.

[1265] Database: A storage system that stores user information, alarm settings, request information, etc.

[1266] AI engines: natural language processing engine, deepfake generation engine, speech synthesis engine, emotion engine

[1267] User Authentication

[1268] The user launches the app and enters their login information. The device encrypts the input information and sends it to the server, which then queries the database to obtain the authentication result. If authentication is successful, the home screen is displayed; if it is unsuccessful, an error message is displayed on the device. This uses data encryption using AES (Advanced Encryption Standard) and authentication using OAuth tokens.

[1269] Conversation function using emotion recognition

[1270] The user selects the conversation function and inputs a message by voice or text. The device recognizes the voice input and converts it into text. This process uses the Google Cloud Speech-to-Text API. The converted text data is sent to the server, where a natural language processing engine (e.g., OpenAI's GPT-3) generates an appropriate response. The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotions and reflect them in the response. Finally, the server uses a deepfake generation engine (e.g., DeepFaceLab) and a speech synthesis engine (e.g., Amazon Polly) to generate video and audio, which are then sent to the device for playback.

[1271] Alarm setting using emotion recognition

[1272] The user opens the alarm settings screen and enters the desired alarm time and wake-up message. The device sends the setting information to the server, which stores the data in a database. When the alarm time comes, the server retrieves the corresponding alarm data and uses an emotion engine to generate deepfake video and audio that matches the user's emotions. The generated data is sent to the device, which then plays the video and audio along with the alarm. DeepFaceLab and Amazon Polly are used in this process.

[1273] Song Request Using Emotion Recognition

[1274] The user inputs a song request, and the device sends the request information to the server. The server then checks the database for copyright information. If the copyright is permitted, the server uses an emotion engine to recognize the user's emotions and generate deepfake video and audio. This uses DeepFaceLab and Amazon Polly. Finally, the generated data is sent to the device, where the user can enjoy the video and audio.

[1275] Specific examples

[1276] Example 1: Conversation function

[1277] When a user launches the app and logs in, the home screen appears, and they tap the conversation function icon to say, "Hello, how are you today?" The device converts the speech into text and sends it to the server. The server analyzes the text and generates a response such as, "I'm feeling good today, how about you?" The emotion engine recognizes emotions from the user's voice and reflects this in the response. The deepfake generation engine generates video that matches the user's emotions, and the speech synthesis engine adjusts the tone to generate the audio. The device plays back the video and audio, and the "oshi" character says, "I'm feeling good today, how about you?"

[1278] Example 2: Alarm settings

[1279] The user opens the alarm settings and sets it to "Good morning, it's time to wake up" at 7:00. When the alarm time comes, the server receives the message "Good morning, it's time to wake up" and uses the emotion engine to generate video and audio that matches the user's emotion. While the device sounds the alarm, it plays video and audio that matches the emotion, and the "oshi" wakes the user up.

[1280] Prompt Sentence Examples

[1281] "Simulate a scenario where a user uses emotion-aware conversation features."

[1282] "Please explain the steps a user goes through to set up an alarm using deepfake technology."

[1283] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1284] User Authentication

[1285] Step 1:

[1286] A user launches the app and accesses the login screen, where they enter their username and password.

[1287] Input: Username, Password

[1288] Output: User credentials

[1289] Step 2:

[1290] The terminal encrypts the entered username and password using AES (Advanced Encryption Standard) and sends an HTTP request.

[1291] Input: User credentials

[1292] Output: Encrypted user credentials

[1293] Step 3:

[1294] The server receives the encrypted user authentication information and queries the database to retrieve the corresponding user information.

[1295] Input: Encrypted user credentials

[1296] Output: Database query results

[1297] Step 4:

[1298] The server returns the authentication result (success or failure) in JSON format, including the home screen data for successful authentication.

[1299] Input: Database query results

[1300] Output: Authentication result (JSON format)

[1301] Step 5:

[1302] The device analyzes the authentication result it receives, and displays the home screen if authentication is successful, or an error message if authentication fails.

[1303] Input: Authentication result (JSON format)

[1304] Output: Home screen or error message

[1305] Conversation function using emotion recognition

[1306] Step 1:

[1307] The user selects the app's conversation feature and enters a message by voice or text.

[1308] Input: Voice or text message

[1309] Output: User message

[1310] Step 2:

[1311] The device converts voice input into text data using the Google Cloud Speech-to-Text API (when using voice input).

[1312] Input: Voice message

[1313] Output: Text data

[1314] Step 3:

[1315] The terminal converts the converted text data into JSON format and sends it to the server via the REST API.

[1316] Input: Text data

[1317] Output: JSON format text data

[1318] Step 4:

[1319] The server receives the text data and passes it to a natural language processing engine (e.g., OpenAI's GPT-3) for analysis.

[1320] Input: JSON format text data

[1321] Output: Response text

[1322] Step 5:

[1323] The natural language processing engine generates appropriate response text and returns it to the server.

[1324] Input: JSON format text data

[1325] Output: Response text

[1326] Step 6:

[1327] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize emotions from the user's text data.

[1328] Input: Response text

[1329] Output: Emotional information

[1330] Step 7:

[1331] Based on the emotional data recognized by the emotion engine, the appropriate emotion is reflected in the response text.

[1332] Input: Emotion information

[1333] Output: Sentiment-reflecting response text

[1334] Step 8:

[1335] The server generates video and audio using a deepfake generation engine (e.g., DeepFaceLab) and a speech synthesis engine (e.g., Amazon Polly).

[1336] Input: Sentiment-reflecting response text

[1337] Output: Video and audio data

[1338] Step 9:

[1339] The server encodes the generated video and audio data and sends it to the terminal as an HTTP response.

[1340] Input: Video and audio data

[1341] Output: Encoded video and audio data

[1342] Step 10:

[1343] The device decodes and plays the received video and audio data using a media player such as VLC Player.

[1344] Input: Encoded video and audio data

[1345] Output: Video and audio playback

[1346] Alarm setting using emotion recognition

[1347] Step 1:

[1348] The user opens the alarm setting screen and enters the desired alarm time and wake-up message.

[1349] Input: Alarm time and wake-up message

[1350] Output: Alarm setting information

[1351] Step 2:

[1352] The terminal converts the input alarm information into JSON format and sends it to the server.

[1353] Input: Alarm setting information

[1354] Output: Alarm configuration information in JSON format

[1355] Step 3:

[1356] The server stores the received alarm information in the database using an INSERT query.

[1357] Input: Alarm setting information in JSON format

[1358] Output: Database saved results

[1359] Step 4:

[1360] When the alarm time arrives, the server queries the corresponding alarm setting data.

[1361] Input: Database saved results (alarm time)

[1362] Output: Alarm setting data

[1363] Step 5:

[1364] The server uses an emotion engine to generate deepfake video and audio that matches the user's emotions.

[1365] Input: Alarm setting data

[1366] Output: Emotion-reflecting video and audio data

[1367] Step 6:

[1368] The server encodes the generated video and audio data and sends it to the terminal in an HTTP response.

[1369] Input: Emotion-reflecting video and audio data

[1370] Output: Encoded video and audio data

[1371] Step 7:

[1372] The terminal detects that the alarm time has come and plays the received video and audio data along with the alarm sound.

[1373] Input: Encoded video and audio data

[1374] Output: Alarm sound and video and audio playback

[1375] Song Request Using Emotion Recognition

[1376] Step 1:

[1377] The user opens the app's song request function and enters the song title and artist name.

[1378] Input: Song title and artist name

[1379] Output: Request information

[1380] Step 2:

[1381] The terminal converts the request information into JSON format and sends it to the server.

[1382] Input: Request information

[1383] Output: Request information in JSON format

[1384] Step 3:

[1385] The server queries the database for copyright information for the requested song, and if necessary consults external copyright databases via an API.

[1386] Input: Request information in JSON format

[1387] Output: Copyright information

[1388] Step 4:

[1389] If copyright permits, the server uses an emotion engine to recognize the user's emotions and generate deepfake video and audio, using DeepFaceLab and Amazon Polly.

[1390] Input: Copyright information

[1391] Output: Emotion-reflecting video and audio data

[1392] Step 5:

[1393] The server encodes the generated video and audio data and sends it to the terminal in an HTTP response.

[1394] Input: Emotion-reflecting video and audio data

[1395] Output: Encoded video and audio data

[1396] Step 6:

[1397] The device receives the data and plays the video and audio using a VLC player or similar, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[1398] Input: Encoded video and audio data

[1399] Output: Video and audio playback

[1400] (Application example 2)

[1401] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1402] Conventional data processing technologies have limited two-way communication with users, making it difficult to provide a personalized experience that reflects the user's emotions in real time. In particular, playback of video and audio content is not adjusted according to the user's emotional state, resulting in a lack of immersive and personalized entertainment. Therefore, there is a need to improve user satisfaction and provide a more immersive viewing experience.

[1403] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving authentication information entered by the user, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting input data from the user into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for generating deepfake video and audio data based on the response, means for transmitting the generated video and audio data to the user terminal and playing it, and means for adjusting the deepfake video and audio data based on the user's emotions using an emotion recognition engine that recognizes the user's emotions. This enables a personalized viewing experience according to the user's emotions.

[1404] The "data processing device" refers to the entire system for realizing two-way communication with the user.

[1405] "Authentication information" refers to information such as a username and password required for a user to access an application.

[1406] A "database" is a storage device for saving and managing data such as user information and request information.

[1407] The "home screen" is the main screen that is displayed when the user is successfully authenticated.

[1408] An "error message" is an error message that is displayed when a user fails authentication.

[1409] "Speech recognition" is the technique or process of analyzing and converting spoken input from a user into text data.

[1410] "Natural language processing" is a technology for analyzing text data from users and generating appropriate responses.

[1411] "Deepfake video" is a synthetic video generated using AI technology that has similar qualities to real-life video.

[1412] "Voice data" refers to data that includes a voice response to a user, and is generated by a voice synthesis engine.

[1413] An "emotion recognition engine" is a technology that analyzes a user's emotions from their facial expressions, voice, etc., and generates an appropriate response based on the results.

[1414] A "user terminal" is a device that is directly operated by a user, such as a smartphone, tablet, or PC.

[1415] This invention is a system that uses a data processing device to realize two-way communication with a user. Specifically, it utilizes an emotion recognition engine, a deepfake generation engine, and a voice synthesis engine to provide a personalized experience based on the user's emotions.

[1416] System Configuration

[1417] The system consists of the following main components:

[1418] 1. User Device:

[1419] These devices include smartphones, tablets, and PCs. These devices are operated by users and are equipped with cameras and microphones.

[1420] 2. Server:

[1421] Performs data processing and generation processes: receives data from user terminals and generates appropriate responses.

[1422] 3. Database:

[1423] Store and manage data such as user information, alarm settings, and request information.

[1424] 4. AI Engine:

[1425] It includes a natural language processing engine, a deepfake generation engine, a speech synthesis engine, and an emotion recognition engine.

[1426] Program Description

[1427] The whole system works in the following steps:

[1428] 1. User authentication:

[1429] The user accesses the application using their device and enters their authentication information. The device sends this information to the server, which then queries the database to obtain the authentication result. Depending on the result, the home screen or an error message is displayed.

[1430] 2. Emotion recognition:

[1431] When a user accesses the system while standing in front of the camera, the emotion recognition engine analyzes the user's facial expressions and voice to recognize their emotions.

[1432] 3. Deepfake Generation:

[1433] Based on the recognized emotional data, the natural language processing engine generates appropriate response text, and based on that, the deepfake generation engine and speech synthesis engine generate video and audio.

[1434] 4. Content playback:

[1435] The generated deepfake video and audio are then sent to the user's device and played back, allowing the user to enjoy a personalized viewing experience based on their emotions.

[1436] Specific examples

[1437] Example 1: Conversation function

[1438] The user launches the application and enters their login information for authentication. After successful authentication, the home screen is displayed.

[1439] The user selects the conversation function and types "How are you today?" The system converts this into text data through voice recognition and sends it to the server.

[1440] The server uses an emotion recognition engine to analyze the user's emotions, and a natural language processing engine generates a response such as "I'm feeling good today, how about you?"

[1441] Based on the emotional data of the response, the deepfake generation engine and voice synthesis engine generate video and audio, which are then sent to the user's device.

[1442] The user terminal plays the generated video and audio, providing the user with a real-time interactive communication experience.

[1443] Example 2: Alarm settings

[1444] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[1445] When the alarm time arrives, the server retrieves the relevant data and uses an emotion recognition engine to generate deepfake video and audio that matches the user's emotions.

[1446] The user terminal sounds an alarm with the generated video and audio, waking the user up in an emotionally appropriate way.

[1447] Prompt Sentence Examples

[1448] "Please provide step-by-step instructions showing how to play personalized video and audio that feels natural to the user. Include instructions for adjusting video and audio presentation based on the user's emotions."

[1449] In this way, the system can provide a personalized viewing experience according to the user's emotions.

[1450] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1451] Step 1:

[1452] A user starts an application and enters login information (username, password). The terminal receives this login information and sends it to the server to query the database. Based on this input, the server looks up the authentication database and returns the result (authentication success or failure).

[1453] Step 2:

[1454] The server queries the database and sends information on whether authentication was successful or not to the device. If authentication was successful, the home screen is displayed, and if it was not, an error message is displayed. This display guides the user on what to do next.

[1455] Step 3:

[1456] The user selects the conversation function from the home screen and inputs either voice or text. In the case of voice input, the device uses a voice recognition API to convert the voice data into text data and sends it to the server. The server receives this text data and analyzes it.

[1457] Step 4:

[1458] The server sends the received text data to the natural language processing engine and begins analysis. The natural language processing engine generates an appropriate response text based on the user's input. This analysis and response generation is performed to output an accurate response to the user's question or request.

[1459] Step 5:

[1460] The server sends the generated response text and data for recognizing the user's emotions to the emotion recognition engine. The emotion recognition engine recognizes the user's emotions (e.g., joy, anger, sadness, and happiness) from the input audio or video data and outputs the results.

[1461] Step 6:

[1462] Based on the emotion recognition results, the server sends data to a deepfake generation engine and a speech synthesis engine to generate response video and audio. During this generation process, the facial expression of the video and the tone of the voice are adjusted based on the emotion information. This procedure provides the user with a response that is emotionally relevant.

[1463] Step 7:

[1464] The server then transmits the generated video and audio data to the user's device, where it is played back. The user receives personalized responses in real time using both video and audio, creating a two-way communication experience. This playback process allows the user to enjoy enhanced visual and audio interactions.

[1465] Based on these steps, the present invention realizes a technique for recognizing a user's emotions in real time and generating and providing appropriate responses based thereon.

[1466] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1467] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1468] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1469] [Third embodiment]

[1470] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1471] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1472] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1473] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1474] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1475] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1476] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1477] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1478] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1479] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1480] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1481] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1482] This invention is a system that allows users to enjoy interactive communication with their "oshi" using a data processing device. Specifically, it utilizes multimodal generation AI with deepfake technology and voice bots, while taking privacy and copyright protection into consideration.

[1483] System Configuration

[1484] This system consists of a user terminal, a server, a database, and an AI engine (natural language processing engine, deepfake generation engine, and voice synthesis engine). The specific operation of each element is explained below.

[1485] User Authentication

[1486] 1. The user launches the app and enters their login information.

[1487] 2. The device sends the login information to the server.

[1488] 3. The server queries the database for login information and obtains the authentication result.

[1489] 4. The server sends the user's home screen if authentication is successful, and an error message if authentication fails.

[1490] 5. The device displays the home screen or an error message.

[1491] Conversation feature

[1492] 1. The user selects the conversation feature and enters information by voice or text.

[1493] 2. The device converts voice input into text (when using voice input).

[1494] 3. The terminal sends the entered text data to the server.

[1495] 4. The server analyzes the text data using a natural language processing engine and generates an appropriate response.

[1496] 5. The server generates response video and audio using a deepfake generation engine and a voice synthesis engine.

[1497] 6. The server transmits the generated video and audio data to the user terminal.

[1498] 7. The device plays video and audio, allowing the user to have a conversation with their favorite idol.

[1499] Alarm settings

[1500] 1. The user sets the alarm time and message.

[1501] 2. The device sends the configuration information to the server.

[1502] 3. The server stores the configuration information in a database.

[1503] 4. When the alarm time arrives, the server generates deepfake video and audio and sends them to the user's device.

[1504] 5. The device will play an alarm along with video and audio, and your favorite idol will wake you up.

[1505] Song Requests

[1506] 1. The user enters song request information.

[1507] 2. The device sends the request information to the server.

[1508] 3. The server checks the copyright information for the requested song.

[1509] 4. If copyright is permitted, the server generates deepfake video and audio and sends them to the user's device.

[1510] 5. The device plays the video and audio, and the user enjoys watching their favorite idol sing the song.

[1511] Specific examples

[1512] Example 1: Conversation function

[1513] When a user launches the app and logs in, the home screen appears.

[1514] The user taps the conversation icon and says, "Hello, how are you today?"

[1515] The device converts the speech into text and sends it to the server.

[1516] The server generates a response saying, "How are you today? How are you?", creating deepfake video and audio.

[1517] When the device plays the video and audio, your favorite idol will say, "I'm feeling good today. How about you?"

[1518] Example 2: Alarm settings

[1519] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[1520] When the alarm time comes, the server generates video and audio saying "Good morning, it's time to wake up" and sends it to the device.

[1521] The device will sound an alarm while playing video and audio, and the "oshi" will wake the user up.

[1522] The system of the present invention allows users to enjoy two-way communication with their "oshi" (favorite idols). It also thoroughly protects privacy and copyrights, promoting sound use.

[1523] The processing flow will be explained below.

[1524] Conversation function processing steps

[1525] Step 1:

[1526] The user launches the application and the login screen appears.

[1527] Step 2:

[1528] The user enters login information (username, password) and taps the login button.

[1529] Step 3:

[1530] The terminal sends the entered login information to the server.

[1531] Step 4:

[1532] The server queries the database for login information and authenticates the user.

[1533] Step 5:

[1534] The server returns the authentication result to the device. If authentication is successful, it returns the home screen data, and if it is unsuccessful, it returns an error message.

[1535] Step 6:

[1536] The device displays the home screen or an error message.

[1537] Step 7:

[1538] The user taps the conversation feature icon on the home screen and enters a message by voice or text.

[1539] Step 8:

[1540] The device calls the speech recognition API to convert voice input into text (when voice input is performed).

[1541] Step 9:

[1542] The terminal transmits the text data to the server.

[1543] Step 10:

[1544] The server passes the text data to a natural language processing engine and begins analysis.

[1545] Step 11:

[1546] A natural language processing engine generates appropriate response text.

[1547] Step 12:

[1548] The server passes the response text to the deepfake generation engine and speech synthesis engine.

[1549] Step 13:

[1550] The deepfake generation engine generates the video response, and the speech synthesis engine generates the audio response.

[1551] Step 14:

[1552] The server transmits the generated video and audio data to the terminal.

[1553] Step 15:

[1554] The device plays back the received video and audio data, enabling conversation with the "favorite."

[1555] Alarm configuration process steps

[1556] Step 1:

[1557] The user selects the alarm setting function and displays the setting screen.

[1558] Step 2:

[1559] The user inputs the alarm time and the message they want to wake up with.

[1560] Step 3:

[1561] The terminal sends the set alarm time and message to the server.

[1562] Step 4:

[1563] The server stores the alarm configuration information in a database.

[1564] Step 5:

[1565] When the alarm time arrives, the server retrieves the saved setting data.

[1566] Step 6:

[1567] The server generates deepfake video and audio data for the alarm.

[1568] Step 7:

[1569] The server transmits the generated data to the terminal.

[1570] Step 8:

[1571] The device will sound an alarm and play deepfake video and audio.

[1572] Song request processing steps

[1573] Step 1:

[1574] The user selects the song request function and the request screen is displayed.

[1575] Step 2:

[1576] The user inputs the title of the song they wish to request.

[1577] Step 3:

[1578] The terminal sends the request information to the server.

[1579] Step 4:

[1580] The server checks the copyright information of the requested song against its database.

[1581] Step 5:

[1582] If copyright permits, the server generates the deepfake video and audio data.

[1583] Step 6:

[1584] The server transmits the generated data to the terminal.

[1585] Step 7:

[1586] The device plays back the received video and audio data and provides a video of the "favorite" person singing the song.

[1587] Example 1

[1588] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1589] With current information processing technology, it is difficult for users to enjoy interactive communication with virtual characters in real time. Furthermore, there is a lack of systems that can generate high-quality audio and video while fully considering privacy and copyright protection. This limits the user experience and reduces satisfaction. Furthermore, implementing features such as alarms and song requests is complex, creating a demand for more user-friendly systems.

[1590] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1591] In this invention, the server includes means for receiving authentication information entered by a user, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting the user's input data into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for generating deepfake video and audio data based on the response, means for transmitting the generated video and audio data to a user terminal and playing it back, means for encrypting the data using SSL / TLS, means for understanding the context and generating a response using a natural language processing engine, and means for generating deepfake technology in high resolution and in real time. This allows users to enjoy high-quality two-way communication in real time through audio and video, and also allows them to use the system safely while protecting their privacy and copyright.

[1592] "User" refers to an individual or organization that uses the system.

[1593] "Authentication information" is information for verifying the identity of a user, and includes a username and password.

[1594] A "database" is a system for efficiently storing and managing information.

[1595] "Home screen" refers to the main operation screen that is displayed after a user has successfully logged in.

[1596] An "error message" is a warning or explanatory message that is sent to the user when an error occurs in the system.

[1597] "Speech recognition" is a technology that converts input speech into text data.

[1598] "Text data" is data that is stored or processed as textual information.

[1599] "Natural language processing" is a technology that allows computers to understand and analyze human language.

[1600] "Deepfake" is a technology that uses artificial intelligence to generate or alter video and audio.

[1601] "Audio data" is data that is stored or processed as audio information.

[1602] "Transmitting" is the act of transferring data from one device to another.

[1603] "Playback" refers to the act of outputting stored audio or video in a form that can be confirmed visually or audibly.

[1604] "Encryption" is a technology that converts data according to rules to make it unreadable to third parties.

[1605] An "alarm" is a function that notifies or warns the user at a specific time set by the user.

[1606] "Copyright" refers to the legal right to protect creative works.

[1607] "Request" means a request from a User to take a particular action or provide data.

[1608] The present invention is a system that allows a user to enjoy two-way communication with a virtual character using a data processing device. Specific embodiments of the present invention will be described below.

[1609] System Configuration

[1610] This system consists of a user terminal, a server, a database, and an AI engine (natural language processing engine, deepfake generation engine, and voice synthesis engine).

[1611] User Authentication

[1612] A user launches the app and enters their login information. The device sends the entered login information to the server. The server queries the database for the login information and obtains the authentication result. If authentication is successful, the server generates a home screen and sends it to the device. If authentication fails, the server generates an error message and sends it to the device. The device displays the home screen or the error message.

[1613] Conversation feature

[1614] The user selects the conversation function and inputs voice or text. The device converts the voice input into text (in the case of voice input). The device sends the converted text data to the server. The server analyzes it using a natural language processing engine and generates an appropriate response. The server creates a response video and audio using a deepfake generation engine and a speech synthesis engine. The server sends the generated video and audio data to the user's device. The device plays the video and audio, and the user enjoys a conversation with their "favorite character."

[1615] Alarm settings

[1616] The user opens the alarm settings and enters the alarm time and message to set it. The device sends the setting information to the server. The server saves the setting information in a database. When the alarm time arrives, the server generates deepfake video and audio and sends it to the user's device. The device plays the video and audio along with the alarm time, and the "oshi" wakes the user up.

[1617] Song Requests

[1618] The user inputs song request information on the app. The device sends the request information to the server. The server checks the copyright information of the requested song in a database. If the copyright is permitted, the server generates deepfake video and audio and sends it to the user's device. The device plays the video and audio, allowing the user to enjoy watching their favorite idol sing.

[1619] Hardware and software used

[1620] User devices: smartphones, tablets, PCs

[1621] Server: High-performance computer, cloud server

[1622] Database: SQL database, NoSQL database

[1623] AI engines: natural language processing engines (e.g., BERT), deepfake generation engines, speech synthesis engines (e.g., Tacotron)

[1624] Specific examples

[1625] Example 1: Conversation function

[1626] When a user launches the app and logs in, the home screen appears.

[1627] The user taps the conversation icon and says, "Hello, how are you today?"

[1628] The device converts the speech into text and sends it to the server.

[1629] The server generates a response saying, "How are you today? How are you?", creating deepfake video and audio.

[1630] When the device plays the video and audio, your favorite idol will say, "I'm feeling good today. How about you?"

[1631] Example 2: Alarm settings

[1632] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[1633] When the alarm time comes, the server generates video and audio saying "Good morning, it's time to wake up" and sends it to the device.

[1634] The device will sound an alarm while playing video and audio, and the "oshi" will wake the user up.

[1635] Prompt Sentence Examples

[1636] 1. "What happens when a user selects the conversation feature and types, 'Hello, how are you today?'"

[1637] 2. "What happens when a user sets an alarm for 7:00 and is woken up by a specified message?"

[1638] The system of the present invention allows users to enjoy two-way communication with their "oshi" and ensures thorough privacy and copyright protection.

[1639] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1640] User authentication process steps

[1641] Step 1:

[1642] The user launches the app and enters their username and password on the login screen.

[1643] Input: The username and password entered by the user

[1644] Output: Login information prepared to be sent to the terminal

[1645] Specific behavior: A login form is displayed on the user interface.

[1646] Step 2:

[1647] The terminal sends the entered username and password to the server.

[1648] Input: Login information (username and password)

[1649] Output: Login information sent to the server

[1650] Specific operation: Login information is encrypted with SSL / TLS and sent to the server.

[1651] Step 3:

[1652] The server queries the database for the received login information and searches for corresponding user data.

[1653] Input: Login information sent to the server

[1654] Output: Authentication result from the database

[1655] What happens: A search query is executed against the database using the username and password.

[1656] Step 4:

[1657] The server determines whether the authentication was successful or not based on the results obtained from the database.

[1658] Input: Authentication results from the database

[1659] Output: Home screen data or error message

[1660] Specific operation: If authentication is successful, home screen data based on the user information is generated; if authentication fails, an error message is generated.

[1661] Step 5:

[1662] The terminal displays the home screen data or an error message received from the server.

[1663] Input: Home screen data or error message sent from the server

[1664] Output: The home screen or error message that the user sees

[1665] Specific Behavior: The display screen is updated and an appropriate message is displayed to the user.

[1666] Conversation function processing steps

[1667] Step 1:

[1668] The user selects the conversation function and enters information by voice or text.

[1669] Input: Voice or text input from the user

[1670] Output: Input data received on the terminal

[1671] Specific behavior: The conversation icon is tapped, and an interface for voice or text input appears.

[1672] Step 2:

[1673] The device converts the input voice into text.

[1674] Input: Audio data (in the case of audio input)

[1675] Output: Text data

[1676] Specific operation: The voice recognition engine analyzes the voice data and converts it into text data.

[1677] Step 3:

[1678] The terminal transmits the converted text data to the server.

[1679] Input: Text data (including voice input)

[1680] Output: Text data sent to the server

[1681] Specific operation: The text data is encrypted and sent to the server.

[1682] Step 4:

[1683] The server analyzes the received data using a natural language processing engine and generates an appropriate response.

[1684] Input: Text data sent to the server

[1685] Output: Response text data

[1686] What it does: A natural language processing engine analyzes text data and generates contextual responses.

[1687] Step 5:

[1688] Based on the response generated by the server, video and audio are generated using a deepfake generation engine and a voice synthesis engine.

[1689] Input: Response text data

[1690] Output: Response video and audio data

[1691] Specific operation: The deepfake generation engine generates a response video, and the speech synthesis engine generates a response voice.

[1692] Step 6:

[1693] The server transmits the generated video and audio data to the user terminal.

[1694] Input: Response video and audio data

[1695] Output: Video and audio data sent to the user's device

[1696] What it does: Data is encrypted and efficiently transmitted to the device.

[1697] Step 7:

[1698] The device plays back the video and audio received.

[1699] Input: Video and audio data sent from the server

[1700] Output: The video and audio played to the user

[1701] Specific operation: Video and audio are played, and the user enjoys a conversation with their "favorite."

[1702] Alarm configuration process steps

[1703] Step 1:

[1704] The user opens the alarm settings and sets the alarm time and message.

[1705] Input: Alarm time and message

[1706] Output: Configuration information received on the device

[1707] Specific operation: The alarm setting screen is displayed and the user enters the time and message.

[1708] Step 2:

[1709] The terminal sends the setting information to the server.

[1710] Input: Alarm time and message

[1711] Output: Configuration information sent to the server

[1712] Specific operation: The configuration information is encrypted and sent to the server.

[1713] Step 3:

[1714] The server stores the configuration information in a database.

[1715] Input: Configuration information sent to the server

[1716] Output: Configuration information stored in the database

[1717] Specific behavior: High availability settings are made when data is saved to the database.

[1718] Step 4:

[1719] When the alarm time arrives, the server generates deepfake video and audio and sends them to the user's device.

[1720] Input: Saved alarm time and message

[1721] Output: Generated deepfake video and audio data

[1722] Specific operation: The alarm is triggered at the alarm time, and video and audio are generated and transmitted.

[1723] Step 5:

[1724] When the alarm time comes on the device, it plays video and audio.

[1725] Input: Video and audio data sent from the server

[1726] Output: The video and audio played to the user

[1727] Specific operation: Video and audio are played in sync with the alarm sound, and the "oshi" character wakes up the user.

[1728] Song request processing steps

[1729] Step 1:

[1730] The user enters song request information on the app.

[1731] Input: Song request information

[1732] Output: Request information received on the device

[1733] Specific operation: A request screen is displayed, and the user enters the desired song title and artist name.

[1734] Step 2:

[1735] The terminal sends the request information to the server.

[1736] Input: Request information

[1737] Output: Request information sent to the server

[1738] Specific operation: The request information is encrypted and sent to the server.

[1739] Step 3:

[1740] The server checks the copyright information of the requested song against its database.

[1741] Input: Request information sent to the server

[1742] Output: Copyright information check result

[1743] What it does: Query the database for copyright information for the requested song.

[1744] Step 4:

[1745] If copyright is permitted, the server generates deepfake video and audio and sends them to the user's device.

[1746] Input: Copyright permission confirmation result, requested song information

[1747] Output: Generated deepfake video and audio data

[1748] Specific operation: The generated video and audio are encrypted and sent to the user terminal.

[1749] Step 5:

[1750] The device plays the video and audio, and the user enjoys watching their favorite idol sing.

[1751] Input: Video and audio data sent from the server

[1752] Output: The video and audio played to the user

[1753] Specific operation: The playback screen is displayed and video and audio are played.

[1754] This concludes the detailed explanation of each processing step in this system, which allows users to enjoy high-quality two-way communication in real time.

[1755] (Application example 1)

[1756] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1757] Conventional advertising systems have difficulty providing interactive and effective advertisements tailored to users' interests. There is a particular demand for advertising that utilizes "favorites" that users feel attached to, but current technology does not provide an effective means for achieving this. Furthermore, if real-time responses to user input are required, advanced technology is required to achieve this. An effective system is needed to solve these problems.

[1758] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1759] In this invention, the server includes a means for a user to select an advertisement display, a means for generating advertising video and audio data featuring the user's "oshi" using a deepfake generation engine and a voice bot, a means for delivering and playing the generated advertising video and audio data to the user's terminal, a means for receiving questions and comments from the user, analyzing them with natural language processing means, and generating appropriate responses, and a means for playing the generated responses via the user's "oshi" using the natural language processing means and the voice bot. This allows users to receive responses in real time while enjoying interactive advertising via their "oshi."

[1760] The "means for receiving authentication information entered by a user" refers to a component or system for receiving login information and authentication information entered by a user via a network.

[1761] The "means for querying a database for authentication information and obtaining an authentication result" refers to a process or system for querying a database for received authentication information, verifying its validity, and obtaining an authentication result.

[1762] "Means for displaying the user's home screen or an error message based on the authentication result" refers to an interface or software for displaying an appropriate screen (home screen or an error message) on the user's device depending on the authentication result.

[1763] "Means for recognizing voice input data from a user and converting it into text data" refers to technology or software that converts voice input data into text format.

[1764] The "natural language processing means for analyzing text data and generating an appropriate response" refers to a natural language processing engine or algorithm for analyzing text data and generating an appropriate response.

[1765] A "means for generating deepfake video and audio data based on responses" is a process or system that uses deepfake technology to create video and audio data based on generated responses.

[1766] "Means for transmitting the generated video and audio data to a user terminal and playing it" refers to a device or application for transmitting the generated deepfake video and audio data to a user's device via a network and playing it.

[1767] The "means by which a user can select to display an advertisement" refers to an interface or software that allows a user to select to view an advertisement.

[1768] "Means for generating advertising video and audio data featuring a user's favorite character using a deepfake generation engine and voice bot" refers to an engine or system that makes full use of deepfake technology and voice bots to create advertising video and audio featuring a user's favorite character or idol.

[1769] "Means for delivering and playing the generated advertising video and audio data to a user terminal" refers to a method or apparatus for transmitting the generated advertising video and audio data to a user's device via a network and automatically playing them.

[1770] "Means for receiving questions and comments from users, analyzing them using natural language processing means, and generating appropriate responses" refers to natural language processing technology for receiving questions and comments from users, analyzing them, and creating appropriate responses.

[1771] "Means for using the natural language processing means and a voice bot to play back the generated response through the user's "oshi"" refers to a system or method for using a voice bot to play back a response generated by natural language processing in the voice of the user's "oshi."

[1772] This invention is an interactive advertising display system that uses the user's "oshi" (favorites) and utilizes deepfake technology and voice bots to provide advertisements featuring people special to the user. The following describes how this system can be specifically realized.

[1773] System Configuration

[1774] The system consists of the following main components:

[1775] 1. User device: Mobile device such as smartphone (iOS / Android)

[1776] 2. Server: A cloud-based server (e.g., an AWS EC2 instance)

[1777] 3. Database: A database for data storage and authentication (e.g., MySQL on AWS RDS)

[1778] 4. AI engines: Deepfake generation engines, voice bots, natural language processing engines (e.g., Google Cloud NLP API, TensorFlow, Twilio Voice API)

[1779] User Authentication

[1780] 1. The user launches the app and enters their login information.

[1781] 2. The device sends the login information to the cloud server.

[1782] 3. The server queries the database for login information and obtains the authentication result.

[1783] 4. If the server is successful, it sends the user's home screen; if not, it sends an error message.

[1784] 5. The device displays the home screen or an error message.

[1785] Ad display function

[1786] 1. The user clicks (or taps) the button to view the ad.

[1787] 2. The server uses a deepfake generation engine and a voice bot to generate advertising video and audio data featuring the user's "favorite person."

[1788] 3. The server distributes the generated video and audio data to the user terminal.

[1789] 4. The device plays the advertising video and audio, and the "oshi" delivers the advertisement to the user.

[1790] Interactive Features

[1791] 1. Users can type in questions or comments during the ad.

[1792] 2. The terminal sends the user's input to the server.

[1793] 3. The server analyzes the user's input using a natural language processing engine and generates an appropriate response.

[1794] 4. The server uses deepfakes and a voice bot to generate video and audio data of the "favorite" responding.

[1795] 5. The terminal plays a video and audio response to the user.

[1796] Recommendations

[1797] 1. The server analyzes the user's past behavior data in the database.

[1798] 2. The server selects the most suitable advertisement based on the analysis results.

[1799] 3. The server generates the selected advertisement using a deepfake generation engine and a voice bot and delivers it to the user's device.

[1800] Specific prompt examples

[1801] A prompt that the user types in the app: "How do I use this product?"

[1802] Prompt the server-side AI model: "Please explain in the best possible terms how to use the product the user cares about."

[1803] In this way, the system of the present invention provides a technique that allows for real-time responses while providing interactive advertising via the user's "oshi."

[1804] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1805] Step 1:

[1806] A user launches the app and enters their login information, which includes their user ID and password, and is sent to the device.

[1807] Step 2:

[1808] The terminal sends the received login information to the cloud server. Specifically, the login information is sent to the cloud server using the HTTPS protocol.

[1809] Step 3:

[1810] The server receives the login information and queries a database, which stores the user's authentication information, to verify its validity.

[1811] Step 4:

[1812] The server obtains the authentication result, generates the user's home screen data if successful, or generates an error message if unsuccessful, and sends this information to the device.

[1813] Step 5:

[1814] The device displays the home screen or an error message, using an HTML rendering engine to display the appropriate screen.

[1815] Step 6:

[1816] The user selects to display an ad. A button click is registered as input, and the device sends this information to the server.

[1817] Step 7:

[1818] The server receives the button click information and launches the deepfake generation engine and voice bot. The deepfake generation engine generates video using TensorFlow, and the voice bot generates audio using the Twilio API.

[1819] Step 8:

[1820] The server transmits the generated advertising video and audio data to the user terminal, where the data is transmitted in compressed form and decoded.

[1821] Step 9:

[1822] The device plays the advertisement video and audio, using the device's video and audio players.

[1823] Step 10:

[1824] The user enters questions or comments during the advertisement. The input is text data, which is sent from the terminal to the server.

[1825] Step 11:

[1826] The server receives the user's input and analyzes it using a natural language processing engine (Google Cloud NLP API), where the input text is parsed and the appropriate response data is generated.

[1827] Step 12:

[1828] The server processes the response data using a deepfake generation engine and voice bot, generating video and audio data in which the "favorite" responds.

[1829] Step 13:

[1830] The server transmits the generated response video and audio data to the user terminal, where the data is recompressed and transmitted.

[1831] Step 14:

[1832] The device plays back the response video and audio, allowing the user to receive a response from their "oshi" in real time.

[1833] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1834] System Configuration

[1835] This invention is a system for realizing two-way communication with users using a data processing device. Specifically, it utilizes deepfake technology, a voice bot, and a multimodal generation AI using an emotion engine to realize real-time interaction with users.

[1836] Basic components

[1837] The system consists of the following main components:

[1838] User devices: smartphones, tablets, PCs, etc.

[1839] Server: Data processing and generation processing

[1840] Database: Stores user information, alarm settings, request information, etc.

[1841] AI engines: natural language processing engine, deepfake generation engine, speech synthesis engine, emotion engine

[1842] User Authentication

[1843] 1. The user launches the app and enters their login information (username, password).

[1844] 2. The device sends the input information to the server.

[1845] 3. The server queries the database for login information and obtains the authentication result.

[1846] 4. The server returns the authentication result to the device, and if authentication is successful, the home screen is displayed, and if it is unsuccessful, an error message is displayed.

[1847] 5. The device displays the home screen or an error message.

[1848] Conversation function using emotion recognition

[1849] 1. The user selects the conversation feature and enters information by voice or text.

[1850] 2. The device uses a speech recognition API to convert voice input into text (when voice input is used).

[1851] 3. The terminal sends the entered text data to the server.

[1852] 4. The server passes the text data to the natural language processing engine and begins analysis.

[1853] 5. The natural language processing engine generates the appropriate response text.

[1854] 6. The server uses the emotion engine to recognize emotions from the user's input data.

[1855] 7. The emotional information recognized by the emotion engine is reflected in response generation.

[1856] 8. The server generates a response video and audio using a deepfake generation engine and a voice synthesis engine, adjusting the facial expression and tone of the audio based on the recognized user emotion.

[1857] 9. The server sends the generated video and audio data to the terminal.

[1858] 10. The device plays video and audio, allowing the user to have a conversation with their favorite idol.

[1859] Alarm setting using emotion recognition

[1860] 1. The user sets the alarm time and message.

[1861] 2. The device sends the configuration information to the server.

[1862] 3. The server stores the configuration information in a database.

[1863] 4. When the alarm time arrives, the server retrieves the data and uses an emotion engine to generate deepfake video and audio data that matches the user's emotions.

[1864] 5. The server sends the generated data to the device.

[1865] 6. The device sounds an alarm and plays emotionally appropriate video and audio to wake the user up.

[1866] Song Request Using Emotion Recognition

[1867] 1. The user enters song request information.

[1868] 2. The device sends the request information to the server.

[1869] 3. The server checks the copyright information of the requested song against its database.

[1870] 4. If copyright permits, the server uses an emotion engine to generate deepfake video and audio data based on the user's emotions.

[1871] 5. The server sends the generated data to the device.

[1872] 6. The device plays the video and audio data, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[1873] Specific examples

[1874] Example 1: Conversation function

[1875] When a user launches the app and logs in, the home screen appears.

[1876] The user taps the conversation icon and says, "Hello, how are you today?"

[1877] The device converts the speech into text and sends it to the server.

[1878] The server performs text analysis and generates a response: "I'm fine today, how about you?"

[1879] The emotion engine recognizes emotions from the user's voice and reflects them in the response.

[1880] The deepfake generation engine generates images that match the user's emotions, and the voice synthesis engine adjusts the tone to generate voice.

[1881] The device plays video and audio, and your favorite idol speaks to you, saying, "I'm feeling good today. How about you?"

[1882] Example 2: Alarm settings

[1883] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[1884] When the alarm time comes, the server receives the message "Good morning, it's time to wake up" and uses an emotion engine to generate video and audio that matches the user's emotions.

[1885] The device sounds an alarm and plays video and audio that matches the user's emotions, and the ``oshi'' character wakes the user up.

[1886] The system of the present invention allows users to enjoy real-time interactive communication with their "oshi" and provides a personalized experience based on their emotions. It also thoroughly protects privacy and copyrights and promotes the healthy use of technology.

[1887] The processing flow will be explained below.

[1888] Processing steps for conversational functions using emotion recognition

[1889] Step 1:

[1890] The user launches the application and the login screen appears.

[1891] Step 2:

[1892] The user enters login information (username, password) and taps the login button.

[1893] Step 3:

[1894] The terminal sends the entered login information to the server.

[1895] Step 4:

[1896] The server queries the database for login information and authenticates the user.

[1897] Step 5:

[1898] The server returns the authentication result to the device. If authentication is successful, it returns the home screen data, and if it is unsuccessful, it returns an error message.

[1899] Step 6:

[1900] The device displays the home screen or an error message.

[1901] Step 7:

[1902] The user taps the conversation feature icon and enters a message by voice or text.

[1903] Step 8:

[1904] The device calls the speech recognition API to convert voice input into text (when voice input is performed).

[1905] Step 9:

[1906] The terminal transmits the text data to the server.

[1907] Step 10:

[1908] The server passes the text data to a natural language processing engine and begins analysis.

[1909] Step 11:

[1910] A natural language processing engine generates appropriate response text.

[1911] Step 12:

[1912] The server uses an emotion engine to recognize emotions from the user's input data.

[1913] Step 13:

[1914] The emotional information recognized by the emotion engine is reflected in response generation.

[1915] Step 14:

[1916] The server uses a deepfake generation engine and a voice synthesis engine to generate a response video and audio, adjusting the facial expression and tone of the audio based on the user's recognized emotions.

[1917] Step 15:

[1918] The server transmits the generated video and audio data to the terminal.

[1919] Step 16:

[1920] The device plays back the received video and audio data, enabling conversation with the "favorite."

[1921] Processing steps for setting alarms using emotion recognition

[1922] Step 1:

[1923] The user selects the alarm setting function and displays the setting screen.

[1924] Step 2:

[1925] The user inputs the alarm time and the message they want to wake up with.

[1926] Step 3:

[1927] The terminal sends the set alarm time and message to the server.

[1928] Step 4:

[1929] The server stores the alarm configuration information in a database.

[1930] Step 5:

[1931] When the alarm time arrives, the server acquires the set alarm information.

[1932] Step 6:

[1933] The server uses an emotion engine to generate deepfake video and audio data for the alarm based on the user's emotions.

[1934] Step 7:

[1935] The server transmits the generated video and audio data to the user terminal.

[1936] Step 8:

[1937] The device sounds an alarm and plays video and audio that matches the emotion.

[1938] Steps for processing song requests using emotion recognition

[1939] Step 1:

[1940] The user selects the song request function and the request screen is displayed.

[1941] Step 2:

[1942] The user inputs the title of the song they wish to request.

[1943] Step 3:

[1944] The terminal sends the request information to the server.

[1945] Step 4:

[1946] The server checks the copyright information of the requested song against its database.

[1947] Step 5:

[1948] If copyright permits, the server uses an emotion engine to generate deepfake video and audio data based on the user's emotions.

[1949] Step 6:

[1950] The server transmits the generated video and audio data to the user terminal.

[1951] Step 7:

[1952] The terminal plays back the received video and audio data, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[1953] Example 2

[1954] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1955] Although internet-based interactive communication technologies have become increasingly sophisticated in recent years, systems that recognize users' emotions in real time and generate personalized video and audio accordingly have yet to become widespread. Furthermore, to improve the user experience, there is a growing demand for systems that seamlessly integrate authentication processes, conversation functions, alarm settings, and song request functions in a way that responds to emotions. However, existing systems struggle to integrate multiple technologies and engines and provide interactions that respond to users' emotions. Furthermore, the processing efficiency and speed of data transmission and reception required to execute these functions in real time remain challenges.

[1956] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user authentication information, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting input data from the user into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for recognizing emotions from the user's input data, means for generating a response reflecting the recognized emotion, means for generating deepfake video and audio data based on the response, and means for transmitting the generated video and audio data to a user terminal and playing them. This enables personalized communication in real time according to the user's emotions. Furthermore, the seamless integration of multiple functions such as authentication, conversation, alarm setting, and song request can significantly improve the user experience.

[1957] "User authentication" is the process in which the server queries a database to verify the user's identity based on the authentication information entered by the user.

[1958] "Speech recognition" is a technology that analyzes voice data entered by a user and converts it into text data.

[1959] "Natural language processing" is a technology for analyzing input text data and generating appropriate responses.

[1960] "Emotion recognition" is a technology that extracts emotions from user input data and recognizes those emotions as data.

[1961] "Deepfake" is a technology that uses artificial intelligence to synthesize events and actions that do not actually exist into video data.

[1962] "Speech synthesis" is a technology that generates voice data based on generated text data.

[1963] A "user terminal" is a device that a user directly operates, and includes smartphones, tablets, PCs, etc.

[1964] A "database" is a system for storing user information, alarm settings, request information, and the like.

[1965] A "server" is a central computer device in a system that processes data and performs production processing.

[1966] "Authentication result" is data indicating whether user authentication was successful or unsuccessful as a result of the server querying the database.

[1967] A "home screen" is the initial interface that is displayed after a user is successfully authenticated.

[1968] An "error message" is a warning message that is displayed when user authentication fails.

[1969] "Response generation" is the process of constructing an appropriate reply to user input using natural language processing and emotion recognition.

[1970] "Video and audio data" refers to multimedia data generated using deepfake and voice synthesis technology.

[1971] "Real-time" refers to responding immediately to user input and executing processing without delay.

[1972] "Alarm settings" is a feature that allows users to set up messages and notifications that will be triggered at specific times.

[1973] A "song request" is the process by which a user requests that a particular song be played.

[1974] "Copyright Information" is data used to verify that the requested song is legally licensed.

[1975] MODE FOR CARRYING OUT THE INVENTION

[1976] System Configuration

[1977] This invention is a system for realizing two-way communication with users using a data processing device. This system utilizes multimodal generation AI with deepfake technology, a voice bot, and an emotion engine, enabling real-time interaction with users.

[1978] Basic components

[1979] The system consists of the following main components:

[1980] User devices: devices such as smartphones, tablets, and PCs

[1981] Server: A computer system responsible for data processing and generation.

[1982] Database: A storage system that stores user information, alarm settings, request information, etc.

[1983] AI engines: natural language processing engine, deepfake generation engine, speech synthesis engine, emotion engine

[1984] User Authentication

[1985] The user launches the app and enters their login information. The device encrypts the input information and sends it to the server, which then queries the database to obtain the authentication result. If authentication is successful, the home screen is displayed; if it is unsuccessful, an error message is displayed on the device. This uses data encryption using AES (Advanced Encryption Standard) and authentication using OAuth tokens.

[1986] Conversation function using emotion recognition

[1987] The user selects the conversation function and inputs a message by voice or text. The device recognizes the voice input and converts it into text. This process uses the Google Cloud Speech-to-Text API. The converted text data is sent to the server, where a natural language processing engine (e.g., OpenAI's GPT-3) generates an appropriate response. The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotions and reflect them in the response. Finally, the server uses a deepfake generation engine (e.g., DeepFaceLab) and a speech synthesis engine (e.g., Amazon Polly) to generate video and audio, which are then sent to the device for playback.

[1988] Alarm setting using emotion recognition

[1989] The user opens the alarm settings screen and enters the desired alarm time and wake-up message. The device sends the setting information to the server, which stores the data in a database. When the alarm time comes, the server retrieves the corresponding alarm data and uses an emotion engine to generate deepfake video and audio that matches the user's emotions. The generated data is sent to the device, which then plays the video and audio along with the alarm. DeepFaceLab and Amazon Polly are used in this process.

[1990] Song Request Using Emotion Recognition

[1991] The user inputs a song request, and the device sends the request information to the server. The server then checks the database for copyright information. If the copyright is permitted, the server uses an emotion engine to recognize the user's emotions and generate deepfake video and audio. This uses DeepFaceLab and Amazon Polly. Finally, the generated data is sent to the device, where the user can enjoy the video and audio.

[1992] Specific examples

[1993] Example 1: Conversation function

[1994] When a user launches the app and logs in, the home screen appears, and they tap the conversation function icon to say, "Hello, how are you today?" The device converts the speech into text and sends it to the server. The server analyzes the text and generates a response such as, "I'm feeling good today, how about you?" The emotion engine recognizes emotions from the user's voice and reflects this in the response. The deepfake generation engine generates video that matches the user's emotions, and the speech synthesis engine adjusts the tone to generate the audio. The device plays back the video and audio, and the "oshi" character says, "I'm feeling good today, how about you?"

[1995] Example 2: Alarm settings

[1996] The user opens the alarm settings and sets it to "Good morning, it's time to wake up" at 7:00. When the alarm time comes, the server receives the message "Good morning, it's time to wake up" and uses the emotion engine to generate video and audio that matches the user's emotion. While the device sounds the alarm, it plays video and audio that matches the emotion, and the "oshi" wakes the user up.

[1997] Prompt Sentence Examples

[1998] "Simulate a scenario where a user uses emotion-aware conversation features."

[1999] "Please explain the steps a user goes through to set up an alarm using deepfake technology."

[2000] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2001] User Authentication

[2002] Step 1:

[2003] A user launches the app and accesses the login screen, where they enter their username and password.

[2004] Input: Username, Password

[2005] Output: User credentials

[2006] Step 2:

[2007] The terminal encrypts the entered username and password using AES (Advanced Encryption Standard) and sends an HTTP request.

[2008] Input: User credentials

[2009] Output: Encrypted user credentials

[2010] Step 3:

[2011] The server receives the encrypted user authentication information and queries the database to retrieve the corresponding user information.

[2012] Input: Encrypted user credentials

[2013] Output: Database query results

[2014] Step 4:

[2015] The server returns the authentication result (success or failure) in JSON format, including the home screen data for successful authentication.

[2016] Input: Database query results

[2017] Output: Authentication result (JSON format)

[2018] Step 5:

[2019] The device analyzes the authentication result it receives, and displays the home screen if authentication is successful, or an error message if authentication fails.

[2020] Input: Authentication result (JSON format)

[2021] Output: Home screen or error message

[2022] Conversation function using emotion recognition

[2023] Step 1:

[2024] The user selects the app's conversation feature and enters a message by voice or text.

[2025] Input: Voice or text message

[2026] Output: User message

[2027] Step 2:

[2028] The device converts voice input into text data using the Google Cloud Speech-to-Text API (when using voice input).

[2029] Input: Voice message

[2030] Output: Text data

[2031] Step 3:

[2032] The terminal converts the converted text data into JSON format and sends it to the server via the REST API.

[2033] Input: Text data

[2034] Output: JSON format text data

[2035] Step 4:

[2036] The server receives the text data and passes it to a natural language processing engine (e.g., OpenAI's GPT-3) for analysis.

[2037] Input: JSON format text data

[2038] Output: Response text

[2039] Step 5:

[2040] The natural language processing engine generates appropriate response text and returns it to the server.

[2041] Input: JSON format text data

[2042] Output: Response text

[2043] Step 6:

[2044] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize emotions from the user's text data.

[2045] Input: Response text

[2046] Output: Emotional information

[2047] Step 7:

[2048] Based on the emotional data recognized by the emotion engine, the appropriate emotion is reflected in the response text.

[2049] Input: Emotion information

[2050] Output: Sentiment-reflecting response text

[2051] Step 8:

[2052] The server generates video and audio using a deepfake generation engine (e.g., DeepFaceLab) and a speech synthesis engine (e.g., Amazon Polly).

[2053] Input: Sentiment-reflecting response text

[2054] Output: Video and audio data

[2055] Step 9:

[2056] The server encodes the generated video and audio data and sends it to the terminal as an HTTP response.

[2057] Input: Video and audio data

[2058] Output: Encoded video and audio data

[2059] Step 10:

[2060] The device decodes and plays the received video and audio data using a media player such as VLC Player.

[2061] Input: Encoded video and audio data

[2062] Output: Video and audio playback

[2063] Alarm setting using emotion recognition

[2064] Step 1:

[2065] The user opens the alarm setting screen and enters the desired alarm time and wake-up message.

[2066] Input: Alarm time and wake-up message

[2067] Output: Alarm setting information

[2068] Step 2:

[2069] The terminal converts the input alarm information into JSON format and sends it to the server.

[2070] Input: Alarm setting information

[2071] Output: Alarm configuration information in JSON format

[2072] Step 3:

[2073] The server stores the received alarm information in the database using an INSERT query.

[2074] Input: Alarm setting information in JSON format

[2075] Output: Database saved results

[2076] Step 4:

[2077] When the alarm time arrives, the server queries the corresponding alarm setting data.

[2078] Input: Database saved results (alarm time)

[2079] Output: Alarm setting data

[2080] Step 5:

[2081] The server uses an emotion engine to generate deepfake video and audio that matches the user's emotions.

[2082] Input: Alarm setting data

[2083] Output: Emotion-reflecting video and audio data

[2084] Step 6:

[2085] The server encodes the generated video and audio data and sends it to the terminal in an HTTP response.

[2086] Input: Emotion-reflecting video and audio data

[2087] Output: Encoded video and audio data

[2088] Step 7:

[2089] The terminal detects that the alarm time has come and plays the received video and audio data along with the alarm sound.

[2090] Input: Encoded video and audio data

[2091] Output: Alarm sound and video and audio playback

[2092] Song Request Using Emotion Recognition

[2093] Step 1:

[2094] The user opens the app's song request function and enters the song title and artist name.

[2095] Input: Song title and artist name

[2096] Output: Request information

[2097] Step 2:

[2098] The terminal converts the request information into JSON format and sends it to the server.

[2099] Input: Request information

[2100] Output: Request information in JSON format

[2101] Step 3:

[2102] The server queries the database for copyright information for the requested song, and if necessary consults external copyright databases via an API.

[2103] Input: Request information in JSON format

[2104] Output: Copyright information

[2105] Step 4:

[2106] If copyright permits, the server uses an emotion engine to recognize the user's emotions and generate deepfake video and audio, using DeepFaceLab and Amazon Polly.

[2107] Input: Copyright information

[2108] Output: Emotion-reflecting video and audio data

[2109] Step 5:

[2110] The server encodes the generated video and audio data and sends it to the terminal in an HTTP response.

[2111] Input: Emotion-reflecting video and audio data

[2112] Output: Encoded video and audio data

[2113] Step 6:

[2114] The device receives the data and plays the video and audio using a VLC player or similar, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[2115] Input: Encoded video and audio data

[2116] Output: Video and audio playback

[2117] (Application example 2)

[2118] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2119] Conventional data processing technologies have limited two-way communication with users, making it difficult to provide a personalized experience that reflects the user's emotions in real time. In particular, playback of video and audio content is not adjusted according to the user's emotional state, resulting in a lack of immersive and personalized entertainment. Therefore, there is a need to improve user satisfaction and provide a more immersive viewing experience.

[2120] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving authentication information entered by the user, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting input data from the user into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for generating deepfake video and audio data based on the response, means for transmitting the generated video and audio data to the user terminal and playing it, and means for adjusting the deepfake video and audio data based on the user's emotions using an emotion recognition engine that recognizes the user's emotions. This enables a personalized viewing experience according to the user's emotions.

[2121] The "data processing device" refers to the entire system for realizing two-way communication with the user.

[2122] "Authentication information" refers to information such as a username and password required for a user to access an application.

[2123] A "database" is a storage device for saving and managing data such as user information and request information.

[2124] The "home screen" is the main screen that is displayed when the user is successfully authenticated.

[2125] An "error message" is an error message that is displayed when a user fails authentication.

[2126] "Speech recognition" is the technique or process of analyzing and converting spoken input from a user into text data.

[2127] "Natural language processing" is a technology for analyzing text data from users and generating appropriate responses.

[2128] "Deepfake video" is a synthetic video generated using AI technology that has similar qualities to real-life video.

[2129] "Voice data" refers to data that includes a voice response to a user, and is generated by a voice synthesis engine.

[2130] An "emotion recognition engine" is a technology that analyzes a user's emotions from their facial expressions, voice, etc., and generates an appropriate response based on the results.

[2131] A "user terminal" is a device that is directly operated by a user, such as a smartphone, tablet, or PC.

[2132] This invention is a system that uses a data processing device to realize two-way communication with a user. Specifically, it utilizes an emotion recognition engine, a deepfake generation engine, and a voice synthesis engine to provide a personalized experience based on the user's emotions.

[2133] System Configuration

[2134] The system consists of the following main components:

[2135] 1. User Device:

[2136] These devices include smartphones, tablets, and PCs. These devices are operated by users and are equipped with cameras and microphones.

[2137] 2. Server:

[2138] Performs data processing and generation processes: receives data from user terminals and generates appropriate responses.

[2139] 3. Database:

[2140] Store and manage data such as user information, alarm settings, and request information.

[2141] 4. AI Engine:

[2142] It includes a natural language processing engine, a deepfake generation engine, a speech synthesis engine, and an emotion recognition engine.

[2143] Program Description

[2144] The whole system works in the following steps:

[2145] 1. User authentication:

[2146] The user accesses the application using their device and enters their authentication information. The device sends this information to the server, which then queries the database to obtain the authentication result. Depending on the result, the home screen or an error message is displayed.

[2147] 2. Emotion recognition:

[2148] When a user accesses the system while standing in front of the camera, the emotion recognition engine analyzes the user's facial expressions and voice to recognize their emotions.

[2149] 3. Deepfake Generation:

[2150] Based on the recognized emotional data, the natural language processing engine generates appropriate response text, and based on that, the deepfake generation engine and speech synthesis engine generate video and audio.

[2151] 4. Content playback:

[2152] The generated deepfake video and audio are then sent to the user's device and played back, allowing the user to enjoy a personalized viewing experience based on their emotions.

[2153] Specific examples

[2154] Example 1: Conversation function

[2155] The user launches the application and enters their login information for authentication. After successful authentication, the home screen is displayed.

[2156] The user selects the conversation function and types "How are you today?" The system converts this into text data through voice recognition and sends it to the server.

[2157] The server uses an emotion recognition engine to analyze the user's emotions, and a natural language processing engine generates a response such as "I'm feeling good today, how about you?"

[2158] Based on the emotional data of the response, the deepfake generation engine and voice synthesis engine generate video and audio, which are then sent to the user's device.

[2159] The user terminal plays the generated video and audio, providing the user with a real-time interactive communication experience.

[2160] Example 2: Alarm settings

[2161] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[2162] When the alarm time arrives, the server retrieves the relevant data and uses an emotion recognition engine to generate deepfake video and audio that matches the user's emotions.

[2163] The user terminal sounds an alarm with the generated video and audio, waking the user up in an emotionally appropriate way.

[2164] Prompt Sentence Examples

[2165] "Please provide step-by-step instructions showing how to play personalized video and audio that feels natural to the user. Include instructions for adjusting video and audio presentation based on the user's emotions."

[2166] In this way, the system can provide a personalized viewing experience according to the user's emotions.

[2167] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2168] Step 1:

[2169] A user starts an application and enters login information (username, password). The terminal receives this login information and sends it to the server to query the database. Based on this input, the server looks up the authentication database and returns the result (authentication success or failure).

[2170] Step 2:

[2171] The server queries the database and sends information on whether authentication was successful or not to the device. If authentication was successful, the home screen is displayed, and if it was not, an error message is displayed. This display guides the user on what to do next.

[2172] Step 3:

[2173] The user selects the conversation function from the home screen and inputs either voice or text. In the case of voice input, the device uses a voice recognition API to convert the voice data into text data and sends it to the server. The server receives this text data and analyzes it.

[2174] Step 4:

[2175] The server sends the received text data to the natural language processing engine and begins analysis. The natural language processing engine generates an appropriate response text based on the user's input. This analysis and response generation is performed to output an accurate response to the user's question or request.

[2176] Step 5:

[2177] The server sends the generated response text and data for recognizing the user's emotions to the emotion recognition engine. The emotion recognition engine recognizes the user's emotions (e.g., joy, anger, sadness, and happiness) from the input audio or video data and outputs the results.

[2178] Step 6:

[2179] Based on the emotion recognition results, the server sends data to a deepfake generation engine and a speech synthesis engine to generate response video and audio. During this generation process, the facial expression of the video and the tone of the voice are adjusted based on the emotion information. This procedure provides the user with a response that is emotionally relevant.

[2180] Step 7:

[2181] The server then transmits the generated video and audio data to the user's device, where it is played back. The user receives personalized responses in real time using both video and audio, creating a two-way communication experience. This playback process allows the user to enjoy enhanced visual and audio interactions.

[2182] Based on these steps, the present invention realizes a technique for recognizing a user's emotions in real time and generating and providing appropriate responses based thereon.

[2183] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2184] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2185] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2186] [Fourth embodiment]

[2187] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2188] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2189] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2190] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2191] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2192] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2193] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2194] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2195] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2196] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2197] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2198] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2199] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2200] This invention is a system that allows users to enjoy interactive communication with their "oshi" using a data processing device. Specifically, it utilizes multimodal generation AI with deepfake technology and voice bots, while taking privacy and copyright protection into consideration.

[2201] System Configuration

[2202] This system consists of a user terminal, a server, a database, and an AI engine (natural language processing engine, deepfake generation engine, and voice synthesis engine). The specific operation of each element is explained below.

[2203] User Authentication

[2204] 1. The user launches the app and enters their login information.

[2205] 2. The device sends the login information to the server.

[2206] 3. The server queries the database for login information and obtains the authentication result.

[2207] 4. The server sends the user's home screen if authentication is successful, and an error message if authentication fails.

[2208] 5. The device displays the home screen or an error message.

[2209] Conversation feature

[2210] 1. The user selects the conversation feature and enters information by voice or text.

[2211] 2. The device converts voice input into text (when using voice input).

[2212] 3. The terminal sends the entered text data to the server.

[2213] 4. The server analyzes the text data using a natural language processing engine and generates an appropriate response.

[2214] 5. The server generates response video and audio using a deepfake generation engine and a voice synthesis engine.

[2215] 6. The server transmits the generated video and audio data to the user terminal.

[2216] 7. The device plays video and audio, allowing the user to have a conversation with their favorite idol.

[2217] Alarm settings

[2218] 1. The user sets the alarm time and message.

[2219] 2. The device sends the configuration information to the server.

[2220] 3. The server stores the configuration information in a database.

[2221] 4. When the alarm time arrives, the server generates deepfake video and audio and sends them to the user's device.

[2222] 5. The device will play an alarm along with video and audio, and your favorite idol will wake you up.

[2223] Song Requests

[2224] 1. The user enters song request information.

[2225] 2. The device sends the request information to the server.

[2226] 3. The server checks the copyright information for the requested song.

[2227] 4. If copyright is permitted, the server generates deepfake video and audio and sends them to the user's device.

[2228] 5. The device plays the video and audio, and the user enjoys watching their favorite idol sing the song.

[2229] Specific examples

[2230] Example 1: Conversation function

[2231] When a user launches the app and logs in, the home screen appears.

[2232] The user taps the conversation icon and says, "Hello, how are you today?"

[2233] The device converts the speech into text and sends it to the server.

[2234] The server generates a response saying, "How are you today? How are you?", creating deepfake video and audio.

[2235] When the device plays the video and audio, your favorite idol will say, "I'm feeling good today. How about you?"

[2236] Example 2: Alarm settings

[2237] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[2238] When the alarm time comes, the server generates video and audio saying "Good morning, it's time to wake up" and sends it to the device.

[2239] The device will sound an alarm while playing video and audio, and the "oshi" will wake the user up.

[2240] The system of the present invention allows users to enjoy two-way communication with their "oshi" (favorite idols). It also thoroughly protects privacy and copyrights, promoting sound use.

[2241] The processing flow will be explained below.

[2242] Conversation function processing steps

[2243] Step 1:

[2244] The user launches the application and the login screen appears.

[2245] Step 2:

[2246] The user enters login information (username, password) and taps the login button.

[2247] Step 3:

[2248] The terminal sends the entered login information to the server.

[2249] Step 4:

[2250] The server queries the database for login information and authenticates the user.

[2251] Step 5:

[2252] The server returns the authentication result to the device. If authentication is successful, it returns the home screen data, and if it is unsuccessful, it returns an error message.

[2253] Step 6:

[2254] The device displays the home screen or an error message.

[2255] Step 7:

[2256] The user taps the conversation feature icon on the home screen and enters a message by voice or text.

[2257] Step 8:

[2258] The device calls the speech recognition API to convert voice input into text (when voice input is performed).

[2259] Step 9:

[2260] The terminal transmits the text data to the server.

[2261] Step 10:

[2262] The server passes the text data to a natural language processing engine and begins analysis.

[2263] Step 11:

[2264] A natural language processing engine generates appropriate response text.

[2265] Step 12:

[2266] The server passes the response text to the deepfake generation engine and speech synthesis engine.

[2267] Step 13:

[2268] The deepfake generation engine generates the video response, and the speech synthesis engine generates the audio response.

[2269] Step 14:

[2270] The server transmits the generated video and audio data to the terminal.

[2271] Step 15:

[2272] The device plays back the received video and audio data, enabling conversation with the "favorite."

[2273] Alarm configuration process steps

[2274] Step 1:

[2275] The user selects the alarm setting function and displays the setting screen.

[2276] Step 2:

[2277] The user inputs the alarm time and the message they want to wake up with.

[2278] Step 3:

[2279] The terminal sends the set alarm time and message to the server.

[2280] Step 4:

[2281] The server stores the alarm configuration information in a database.

[2282] Step 5:

[2283] When the alarm time arrives, the server retrieves the saved setting data.

[2284] Step 6:

[2285] The server generates deepfake video and audio data for the alarm.

[2286] Step 7:

[2287] The server transmits the generated data to the terminal.

[2288] Step 8:

[2289] The device will sound an alarm and play deepfake video and audio.

[2290] Song request processing steps

[2291] Step 1:

[2292] The user selects the song request function and the request screen is displayed.

[2293] Step 2:

[2294] The user inputs the title of the song they wish to request.

[2295] Step 3:

[2296] The terminal sends the request information to the server.

[2297] Step 4:

[2298] The server checks the copyright information of the requested song against its database.

[2299] Step 5:

[2300] If copyright permits, the server generates the deepfake video and audio data.

[2301] Step 6:

[2302] The server transmits the generated data to the terminal.

[2303] Step 7:

[2304] The device plays back the received video and audio data and provides a video of the "favorite" person singing the song.

[2305] Example 1

[2306] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2307] With current information processing technology, it is difficult for users to enjoy interactive communication with virtual characters in real time. Furthermore, there is a lack of systems that can generate high-quality audio and video while fully considering privacy and copyright protection. This limits the user experience and reduces satisfaction. Furthermore, implementing features such as alarms and song requests is complex, creating a demand for more user-friendly systems.

[2308] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2309] In this invention, the server includes means for receiving authentication information entered by a user, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting the user's input data into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for generating deepfake video and audio data based on the response, means for transmitting the generated video and audio data to a user terminal and playing it back, means for encrypting the data using SSL / TLS, means for understanding the context and generating a response using a natural language processing engine, and means for generating deepfake technology in high resolution and in real time. This allows users to enjoy high-quality two-way communication in real time through audio and video, and also allows them to use the system safely while protecting their privacy and copyright.

[2310] "User" refers to an individual or organization that uses the system.

[2311] "Authentication information" is information for verifying the identity of a user, and includes a username and password.

[2312] A "database" is a system for efficiently storing and managing information.

[2313] "Home screen" refers to the main operation screen that is displayed after a user has successfully logged in.

[2314] An "error message" is a warning or explanatory message that is sent to the user when an error occurs in the system.

[2315] "Speech recognition" is a technology that converts input speech into text data.

[2316] "Text data" is data that is stored or processed as textual information.

[2317] "Natural language processing" is a technology that allows computers to understand and analyze human language.

[2318] "Deepfake" is a technology that uses artificial intelligence to generate or alter video and audio.

[2319] "Audio data" is data that is stored or processed as audio information.

[2320] "Transmitting" is the act of transferring data from one device to another.

[2321] "Playback" refers to the act of outputting stored audio or video in a form that can be confirmed visually or audibly.

[2322] "Encryption" is a technology that converts data according to rules to make it unreadable to third parties.

[2323] An "alarm" is a function that notifies or warns the user at a specific time set by the user.

[2324] "Copyright" refers to the legal right to protect creative works.

[2325] "Request" means a request from a User to take a particular action or provide data.

[2326] The present invention is a system that allows a user to enjoy two-way communication with a virtual character using a data processing device. Specific embodiments of the present invention will be described below.

[2327] System Configuration

[2328] This system consists of a user terminal, a server, a database, and an AI engine (natural language processing engine, deepfake generation engine, and voice synthesis engine).

[2329] User Authentication

[2330] A user launches the app and enters their login information. The device sends the entered login information to the server. The server queries the database for the login information and obtains the authentication result. If authentication is successful, the server generates a home screen and sends it to the device. If authentication fails, the server generates an error message and sends it to the device. The device displays the home screen or the error message.

[2331] Conversation feature

[2332] The user selects the conversation function and inputs voice or text. The device converts the voice input into text (in the case of voice input). The device sends the converted text data to the server. The server analyzes it using a natural language processing engine and generates an appropriate response. The server creates a response video and audio using a deepfake generation engine and a speech synthesis engine. The server sends the generated video and audio data to the user's device. The device plays the video and audio, and the user enjoys a conversation with their "favorite character."

[2333] Alarm settings

[2334] The user opens the alarm settings and enters the alarm time and message to set it. The device sends the setting information to the server. The server saves the setting information in a database. When the alarm time arrives, the server generates deepfake video and audio and sends it to the user's device. The device plays the video and audio along with the alarm time, and the "oshi" wakes the user up.

[2335] Song Requests

[2336] The user inputs song request information on the app. The device sends the request information to the server. The server checks the copyright information of the requested song in a database. If the copyright is permitted, the server generates deepfake video and audio and sends it to the user's device. The device plays the video and audio, allowing the user to enjoy watching their favorite idol sing.

[2337] Hardware and software used

[2338] User devices: smartphones, tablets, PCs

[2339] Server: High-performance computer, cloud server

[2340] Database: SQL database, NoSQL database

[2341] AI engines: natural language processing engines (e.g., BERT), deepfake generation engines, speech synthesis engines (e.g., Tacotron)

[2342] Specific examples

[2343] Example 1: Conversation function

[2344] When a user launches the app and logs in, the home screen appears.

[2345] The user taps the conversation icon and says, "Hello, how are you today?"

[2346] The device converts the speech into text and sends it to the server.

[2347] The server generates a response saying, "How are you today? How are you?", creating deepfake video and audio.

[2348] When the device plays the video and audio, your favorite idol will say, "I'm feeling good today. How about you?"

[2349] Example 2: Alarm settings

[2350] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[2351] When the alarm time comes, the server generates video and audio saying "Good morning, it's time to wake up" and sends it to the device.

[2352] The device will sound an alarm while playing video and audio, and the "oshi" will wake the user up.

[2353] Prompt Sentence Examples

[2354] 1. "What happens when a user selects the conversation feature and types, 'Hello, how are you today?'"

[2355] 2. "What happens when a user sets an alarm for 7:00 and is woken up by a specified message?"

[2356] The system of the present invention allows users to enjoy two-way communication with their "oshi" and ensures thorough privacy and copyright protection.

[2357] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2358] User authentication process steps

[2359] Step 1:

[2360] The user launches the app and enters their username and password on the login screen.

[2361] Input: The username and password entered by the user

[2362] Output: Login information prepared to be sent to the terminal

[2363] Specific behavior: A login form is displayed on the user interface.

[2364] Step 2:

[2365] The terminal sends the entered username and password to the server.

[2366] Input: Login information (username and password)

[2367] Output: Login information sent to the server

[2368] Specific operation: Login information is encrypted with SSL / TLS and sent to the server.

[2369] Step 3:

[2370] The server queries the database for the received login information and searches for corresponding user data.

[2371] Input: Login information sent to the server

[2372] Output: Authentication result from the database

[2373] What happens: A search query is executed against the database using the username and password.

[2374] Step 4:

[2375] The server determines whether the authentication was successful or not based on the results obtained from the database.

[2376] Input: Authentication results from the database

[2377] Output: Home screen data or error message

[2378] Specific operation: If authentication is successful, home screen data based on the user information is generated; if authentication fails, an error message is generated.

[2379] Step 5:

[2380] The terminal displays the home screen data or an error message received from the server.

[2381] Input: Home screen data or error message sent from the server

[2382] Output: The home screen or error message that the user sees

[2383] Specific Behavior: The display screen is updated and an appropriate message is displayed to the user.

[2384] Conversation function processing steps

[2385] Step 1:

[2386] The user selects the conversation function and enters information by voice or text.

[2387] Input: Voice or text input from the user

[2388] Output: Input data received on the terminal

[2389] Specific behavior: The conversation icon is tapped, and an interface for voice or text input appears.

[2390] Step 2:

[2391] The device converts the input voice into text.

[2392] Input: Audio data (in the case of audio input)

[2393] Output: Text data

[2394] Specific operation: The voice recognition engine analyzes the voice data and converts it into text data.

[2395] Step 3:

[2396] The terminal transmits the converted text data to the server.

[2397] Input: Text data (including voice input)

[2398] Output: Text data sent to the server

[2399] Specific operation: The text data is encrypted and sent to the server.

[2400] Step 4:

[2401] The server analyzes the received data using a natural language processing engine and generates an appropriate response.

[2402] Input: Text data sent to the server

[2403] Output: Response text data

[2404] What it does: A natural language processing engine analyzes text data and generates contextual responses.

[2405] Step 5:

[2406] Based on the response generated by the server, video and audio are generated using a deepfake generation engine and a voice synthesis engine.

[2407] Input: Response text data

[2408] Output: Response video and audio data

[2409] Specific operation: The deepfake generation engine generates a response video, and the speech synthesis engine generates a response voice.

[2410] Step 6:

[2411] The server transmits the generated video and audio data to the user terminal.

[2412] Input: Response video and audio data

[2413] Output: Video and audio data sent to the user's device

[2414] What it does: Data is encrypted and efficiently transmitted to the device.

[2415] Step 7:

[2416] The device plays back the video and audio received.

[2417] Input: Video and audio data sent from the server

[2418] Output: The video and audio played to the user

[2419] Specific operation: Video and audio are played, and the user enjoys a conversation with their "favorite."

[2420] Alarm configuration process steps

[2421] Step 1:

[2422] The user opens the alarm settings and sets the alarm time and message.

[2423] Input: Alarm time and message

[2424] Output: Configuration information received on the device

[2425] Specific operation: The alarm setting screen is displayed and the user enters the time and message.

[2426] Step 2:

[2427] The terminal sends the setting information to the server.

[2428] Input: Alarm time and message

[2429] Output: Configuration information sent to the server

[2430] Specific operation: The configuration information is encrypted and sent to the server.

[2431] Step 3:

[2432] The server stores the configuration information in a database.

[2433] Input: Configuration information sent to the server

[2434] Output: Configuration information stored in the database

[2435] Specific behavior: High availability settings are made when data is saved to the database.

[2436] Step 4:

[2437] When the alarm time arrives, the server generates deepfake video and audio and sends them to the user's device.

[2438] Input: Saved alarm time and message

[2439] Output: Generated deepfake video and audio data

[2440] Specific operation: The alarm is triggered at the alarm time, and video and audio are generated and transmitted.

[2441] Step 5:

[2442] When the alarm time comes on the device, it plays video and audio.

[2443] Input: Video and audio data sent from the server

[2444] Output: The video and audio played to the user

[2445] Specific operation: Video and audio are played in sync with the alarm sound, and the "oshi" character wakes up the user.

[2446] Song request processing steps

[2447] Step 1:

[2448] The user enters song request information on the app.

[2449] Input: Song request information

[2450] Output: Request information received on the device

[2451] Specific operation: A request screen is displayed, and the user enters the desired song title and artist name.

[2452] Step 2:

[2453] The terminal sends the request information to the server.

[2454] Input: Request information

[2455] Output: Request information sent to the server

[2456] Specific operation: The request information is encrypted and sent to the server.

[2457] Step 3:

[2458] The server checks the copyright information of the requested song against its database.

[2459] Input: Request information sent to the server

[2460] Output: Copyright information check result

[2461] What it does: Query the database for copyright information for the requested song.

[2462] Step 4:

[2463] If copyright is permitted, the server generates deepfake video and audio and sends them to the user's device.

[2464] Input: Copyright permission confirmation result, requested song information

[2465] Output: Generated deepfake video and audio data

[2466] Specific operation: The generated video and audio are encrypted and sent to the user terminal.

[2467] Step 5:

[2468] The device plays the video and audio, and the user enjoys watching their favorite idol sing.

[2469] Input: Video and audio data sent from the server

[2470] Output: The video and audio played to the user

[2471] Specific operation: The playback screen is displayed and video and audio are played.

[2472] This concludes the detailed explanation of each processing step in this system, which allows users to enjoy high-quality two-way communication in real time.

[2473] (Application example 1)

[2474] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2475] Conventional advertising systems have difficulty providing interactive and effective advertisements tailored to users' interests. There is a particular demand for advertising that utilizes "favorites" that users feel attached to, but current technology does not provide an effective means for achieving this. Furthermore, if real-time responses to user input are required, advanced technology is required to achieve this. An effective system is needed to solve these problems.

[2476] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2477] In this invention, the server includes a means for a user to select an advertisement display, a means for generating advertising video and audio data featuring the user's "oshi" using a deepfake generation engine and a voice bot, a means for delivering and playing the generated advertising video and audio data to the user's terminal, a means for receiving questions and comments from the user, analyzing them with natural language processing means, and generating appropriate responses, and a means for playing the generated responses via the user's "oshi" using the natural language processing means and the voice bot. This allows users to receive responses in real time while enjoying interactive advertising via their "oshi."

[2478] The "means for receiving authentication information entered by a user" refers to a component or system for receiving login information and authentication information entered by a user via a network.

[2479] The "means for querying a database for authentication information and obtaining an authentication result" refers to a process or system for querying a database for received authentication information, verifying its validity, and obtaining an authentication result.

[2480] "Means for displaying the user's home screen or an error message based on the authentication result" refers to an interface or software for displaying an appropriate screen (home screen or an error message) on the user's device depending on the authentication result.

[2481] "Means for recognizing voice input data from a user and converting it into text data" refers to technology or software that converts voice input data into text format.

[2482] The "natural language processing means for analyzing text data and generating an appropriate response" refers to a natural language processing engine or algorithm for analyzing text data and generating an appropriate response.

[2483] A "means for generating deepfake video and audio data based on responses" is a process or system that uses deepfake technology to create video and audio data based on generated responses.

[2484] "Means for transmitting the generated video and audio data to a user terminal and playing it" refers to a device or application for transmitting the generated deepfake video and audio data to a user's device via a network and playing it.

[2485] The "means by which a user can select to display an advertisement" refers to an interface or software that allows a user to select to view an advertisement.

[2486] "Means for generating advertising video and audio data featuring a user's favorite character using a deepfake generation engine and voice bot" refers to an engine or system that makes full use of deepfake technology and voice bots to create advertising video and audio featuring a user's favorite character or idol.

[2487] "Means for delivering and playing the generated advertising video and audio data to a user terminal" refers to a method or apparatus for transmitting the generated advertising video and audio data to a user's device via a network and automatically playing them.

[2488] "Means for receiving questions and comments from users, analyzing them using natural language processing means, and generating appropriate responses" refers to natural language processing technology for receiving questions and comments from users, analyzing them, and creating appropriate responses.

[2489] "Means for using the natural language processing means and a voice bot to play back the generated response through the user's "oshi"" refers to a system or method for using a voice bot to play back a response generated by natural language processing in the voice of the user's "oshi."

[2490] This invention is an interactive advertising display system that uses the user's "oshi" (favorites) and utilizes deepfake technology and voice bots to provide advertisements featuring people special to the user. The following describes how this system can be specifically realized.

[2491] System Configuration

[2492] The system consists of the following main components:

[2493] 1. User device: Mobile device such as smartphone (iOS / Android)

[2494] 2. Server: A cloud-based server (e.g., an AWS EC2 instance)

[2495] 3. Database: A database for data storage and authentication (e.g., MySQL on AWS RDS)

[2496] 4. AI engines: Deepfake generation engines, voice bots, natural language processing engines (e.g., Google Cloud NLP API, TensorFlow, Twilio Voice API)

[2497] User Authentication

[2498] 1. The user launches the app and enters their login information.

[2499] 2. The device sends the login information to the cloud server.

[2500] 3. The server queries the database for login information and obtains the authentication result.

[2501] 4. If the server is successful, it sends the user's home screen; if not, it sends an error message.

[2502] 5. The device displays the home screen or an error message.

[2503] Ad display function

[2504] 1. The user clicks (or taps) the button to view the ad.

[2505] 2. The server uses a deepfake generation engine and a voice bot to generate advertising video and audio data featuring the user's "favorite person."

[2506] 3. The server distributes the generated video and audio data to the user terminal.

[2507] 4. The device plays the advertising video and audio, and the "oshi" delivers the advertisement to the user.

[2508] Interactive Features

[2509] 1. Users can type in questions or comments during the ad.

[2510] 2. The terminal sends the user's input to the server.

[2511] 3. The server analyzes the user's input using a natural language processing engine and generates an appropriate response.

[2512] 4. The server uses deepfakes and a voice bot to generate video and audio data of the "favorite" responding.

[2513] 5. The terminal plays a video and audio response to the user.

[2514] Recommendations

[2515] 1. The server analyzes the user's past behavior data in the database.

[2516] 2. The server selects the most suitable advertisement based on the analysis results.

[2517] 3. The server generates the selected advertisement using a deepfake generation engine and a voice bot and delivers it to the user's device.

[2518] Specific prompt examples

[2519] A prompt that the user types in the app: "How do I use this product?"

[2520] Prompt the server-side AI model: "Please explain in the best possible terms how to use the product the user cares about."

[2521] In this way, the system of the present invention provides a technique that allows for real-time responses while providing interactive advertising via the user's "oshi."

[2522] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2523] Step 1:

[2524] A user launches the app and enters their login information, which includes their user ID and password, and is sent to the device.

[2525] Step 2:

[2526] The terminal sends the received login information to the cloud server. Specifically, the login information is sent to the cloud server using the HTTPS protocol.

[2527] Step 3:

[2528] The server receives the login information and queries a database, which stores the user's authentication information, to verify its validity.

[2529] Step 4:

[2530] The server obtains the authentication result, generates the user's home screen data if successful, or generates an error message if unsuccessful, and sends this information to the device.

[2531] Step 5:

[2532] The device displays the home screen or an error message, using an HTML rendering engine to display the appropriate screen.

[2533] Step 6:

[2534] The user selects to display an ad. A button click is registered as input, and the device sends this information to the server.

[2535] Step 7:

[2536] The server receives the button click information and launches the deepfake generation engine and voice bot. The deepfake generation engine generates video using TensorFlow, and the voice bot generates audio using the Twilio API.

[2537] Step 8:

[2538] The server transmits the generated advertising video and audio data to the user terminal, where the data is transmitted in compressed form and decoded.

[2539] Step 9:

[2540] The device plays the advertisement video and audio, using the device's video and audio players.

[2541] Step 10:

[2542] The user enters questions or comments during the advertisement. The input is text data, which is sent from the terminal to the server.

[2543] Step 11:

[2544] The server receives the user's input and analyzes it using a natural language processing engine (Google Cloud NLP API), where the input text is parsed and the appropriate response data is generated.

[2545] Step 12:

[2546] The server processes the response data using a deepfake generation engine and voice bot, generating video and audio data in which the "favorite" responds.

[2547] Step 13:

[2548] The server transmits the generated response video and audio data to the user terminal, where the data is recompressed and transmitted.

[2549] Step 14:

[2550] The device plays back the response video and audio, allowing the user to receive a response from their "oshi" in real time.

[2551] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2552] System Configuration

[2553] This invention is a system for realizing two-way communication with users using a data processing device. Specifically, it utilizes deepfake technology, a voice bot, and a multimodal generation AI using an emotion engine to realize real-time interaction with users.

[2554] Basic components

[2555] The system consists of the following main components:

[2556] User devices: smartphones, tablets, PCs, etc.

[2557] Server: Data processing and generation processing

[2558] Database: Stores user information, alarm settings, request information, etc.

[2559] AI engines: natural language processing engine, deepfake generation engine, speech synthesis engine, emotion engine

[2560] User Authentication

[2561] 1. The user launches the app and enters their login information (username, password).

[2562] 2. The device sends the input information to the server.

[2563] 3. The server queries the database for login information and obtains the authentication result.

[2564] 4. The server returns the authentication result to the device, and if authentication is successful, the home screen is displayed, and if it is unsuccessful, an error message is displayed.

[2565] 5. The device displays the home screen or an error message.

[2566] Conversation function using emotion recognition

[2567] 1. The user selects the conversation feature and enters information by voice or text.

[2568] 2. The device uses a speech recognition API to convert voice input into text (when voice input is used).

[2569] 3. The terminal sends the entered text data to the server.

[2570] 4. The server passes the text data to the natural language processing engine and begins analysis.

[2571] 5. The natural language processing engine generates the appropriate response text.

[2572] 6. The server uses the emotion engine to recognize emotions from the user's input data.

[2573] 7. The emotional information recognized by the emotion engine is reflected in response generation.

[2574] 8. The server generates a response video and audio using a deepfake generation engine and a voice synthesis engine, adjusting the facial expression and tone of the audio based on the recognized user emotion.

[2575] 9. The server sends the generated video and audio data to the terminal.

[2576] 10. The device plays video and audio, allowing the user to have a conversation with their favorite idol.

[2577] Alarm setting using emotion recognition

[2578] 1. The user sets the alarm time and message.

[2579] 2. The device sends the configuration information to the server.

[2580] 3. The server stores the configuration information in a database.

[2581] 4. When the alarm time arrives, the server retrieves the data and uses an emotion engine to generate deepfake video and audio data that matches the user's emotions.

[2582] 5. The server sends the generated data to the device.

[2583] 6. The device sounds an alarm and plays emotionally appropriate video and audio to wake the user up.

[2584] Song Request Using Emotion Recognition

[2585] 1. The user enters song request information.

[2586] 2. The device sends the request information to the server.

[2587] 3. The server checks the copyright information of the requested song against its database.

[2588] 4. If copyright permits, the server uses an emotion engine to generate deepfake video and audio data based on the user's emotions.

[2589] 5. The server sends the generated data to the device.

[2590] 6. The device plays the video and audio data, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[2591] Specific examples

[2592] Example 1: Conversation function

[2593] When a user launches the app and logs in, the home screen appears.

[2594] The user taps the conversation icon and says, "Hello, how are you today?"

[2595] The device converts the speech into text and sends it to the server.

[2596] The server performs text analysis and generates a response: "I'm fine today, how about you?"

[2597] The emotion engine recognizes emotions from the user's voice and reflects them in the response.

[2598] The deepfake generation engine generates images that match the user's emotions, and the voice synthesis engine adjusts the tone to generate voice.

[2599] The device plays video and audio, and your favorite idol speaks to you, saying, "I'm feeling good today. How about you?"

[2600] Example 2: Alarm settings

[2601] The user opens the alarm settings and sets it for 7:00, saying "Good morning, it's time to wake up."

[2602] When the alarm time comes, the server receives the message "Good morning, it's time to wake up" and uses an emotion engine to generate video and audio that matches the user's emotions.

[2603] The device sounds an alarm and plays video and audio that matches the user's emotions, and the ``oshi'' character wakes the user up.

[2604] The system of the present invention allows users to enjoy real-time interactive communication with their "oshi" and provides a personalized experience based on their emotions. It also thoroughly protects privacy and copyrights and promotes the healthy use of technology.

[2605] The processing flow will be explained below.

[2606] Processing steps for conversational functions using emotion recognition

[2607] Step 1:

[2608] The user launches the application and the login screen appears.

[2609] Step 2:

[2610] The user enters login information (username, password) and taps the login button.

[2611] Step 3:

[2612] The terminal sends the entered login information to the server.

[2613] Step 4:

[2614] The server queries the database for login information and authenticates the user.

[2615] Step 5:

[2616] The server returns the authentication result to the device. If authentication is successful, it returns the home screen data, and if it is unsuccessful, it returns an error message.

[2617] Step 6:

[2618] The device displays the home screen or an error message.

[2619] Step 7:

[2620] The user taps the conversation feature icon and enters a message by voice or text.

[2621] Step 8:

[2622] The device calls the speech recognition API to convert voice input into text (when voice input is performed).

[2623] Step 9:

[2624] The terminal transmits the text data to the server.

[2625] Step 10:

[2626] The server passes the text data to a natural language processing engine and begins analysis.

[2627] Step 11:

[2628] A natural language processing engine generates appropriate response text.

[2629] Step 12:

[2630] The server uses an emotion engine to recognize emotions from the user's input data.

[2631] Step 13:

[2632] The emotional information recognized by the emotion engine is reflected in response generation.

[2633] Step 14:

[2634] The server uses a deepfake generation engine and a voice synthesis engine to generate a response video and audio, adjusting the facial expression and tone of the audio based on the user's recognized emotions.

[2635] Step 15:

[2636] The server transmits the generated video and audio data to the terminal.

[2637] Step 16:

[2638] The device plays back the received video and audio data, enabling conversation with the "favorite."

[2639] Processing steps for setting alarms using emotion recognition

[2640] Step 1:

[2641] The user selects the alarm setting function and displays the setting screen.

[2642] Step 2:

[2643] The user inputs the alarm time and the message they want to wake up with.

[2644] Step 3:

[2645] The terminal sends the set alarm time and message to the server.

[2646] Step 4:

[2647] The server stores the alarm configuration information in a database.

[2648] Step 5:

[2649] When the alarm time arrives, the server acquires the set alarm information.

[2650] Step 6:

[2651] The server uses an emotion engine to generate deepfake video and audio data for the alarm based on the user's emotions.

[2652] Step 7:

[2653] The server transmits the generated video and audio data to the user terminal.

[2654] Step 8:

[2655] The device sounds an alarm and plays video and audio that matches the emotion.

[2656] Steps for processing song requests using emotion recognition

[2657] Step 1:

[2658] The user selects the song request function and the request screen is displayed.

[2659] Step 2:

[2660] The user inputs the title of the song they wish to request.

[2661] Step 3:

[2662] The terminal sends the request information to the server.

[2663] Step 4:

[2664] The server checks the copyright information of the requested song against its database.

[2665] Step 5:

[2666] If copyright permits, the server uses an emotion engine to generate deepfake video and audio data based on the user's emotions.

[2667] Step 6:

[2668] The server transmits the generated video and audio data to the user terminal.

[2669] Step 7:

[2670] The terminal plays back the received video and audio data, allowing the user to enjoy the song of their favorite idol in harmony with their emotions.

[2671] Example 2

[2672] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2673] Although internet-based interactive communication technologies have become increasingly sophisticated in recent years, systems that recognize users' emotions in real time and generate personalized video and audio accordingly have yet to become widespread. Furthermore, to improve the user experience, there is a growing demand for systems that seamlessly integrate authentication processes, conversation functions, alarm settings, and song request functions in a way that responds to emotions. However, existing systems struggle to integrate multiple technologies and engines and provide interactions that respond to users' emotions. Furthermore, the processing efficiency and speed of data transmission and reception required to execute these functions in real time remain challenges.

[2674] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user authentication information, means for querying a database for the authentication information and obtaining an authentication result, means for displaying the user's home screen or an error message based on the authentication result, means for speech recognition and converting input data from the user into text data, natural language processing means for analyzing the text data and generating an appropriate response, means for recognizing emotions from the user's input data, means for generating a response reflecting the recognized emotion, means for generating deepfake video and audio data based on the response, and means for transmitting the generated video and audio data to a user terminal and playing them. This enables personalized communication in real time according to the user's emotions. Furthermore, the seamless integration of multiple functions such as authentication, conversation, alarm setting, and song request can significantly improve the user experience.

[2675] "User authentication" is the process in which the server queries a database to verify the user's identity based on the authentication information entered by the user.

[2676] "Speech recognition" is a technology that analyzes voice data entered by a user and converts it into text data.

[2677] "Natural language processing" is a technology for analyzing input text data and generating appropriate responses.

[2678] "Emotion recognition" is a technology that extracts emotions from user input data and recognizes those emotions as data.

[2679] "Deepfake" is a technology that uses artificial intelligence to synthesize events and actions that do not actually exist into video data.

[2680] "Speech synthesis" is a technology that generates voice data based on generated text data.

[2681] A "user terminal" is a device that a user directly operates, and includes smartphones, tablets, PCs, etc.

[2682] A "database" is a system for storing user information, alarm settings, request information, and the like.

[2683] A "server" is a central computer device in a system that processes data and performs production processing.

[2684] "Authentication result" is data indicating whether user authentication was successful or unsuccessful as a result of the server querying the database.

[2685] A "home screen" is the initial interface that is displayed after a user is successfully authenticated.

[2686] An "error message" is a warning message that is displayed when user authentication fails.

[2687] "Response generation" is the process of constructing an appropriate reply to user input using natural language processing and emotion recognition.

[2688] "Video and audio data" refers to multimedia data generated using deepfake and voice synthesis technology.

[2689] "Real-time" refers to responding immediately to user input and executing processing without delay.

[2690] "Alarm settings" is a feature that allows users to set up messages and notifications that will be triggered at specific times.

[2691] A "song request" is the process by which a user requests that a particular song be played.

[2692] "Copyright Information" is data used to verify that the requested song is legally licensed.

[2693] MODE FOR CARRYING OUT THE INVENTION

[2694] System Configuration

[2695] This invention is a system for realizing two-way communication with users using a data processing device. This system utilizes multimodal generation AI with deepfake technology, a voice bot, and an emotion engine, enabling real-time interaction with users.

[2696] Basic components

[2697] The system consists of the following main components:

[2698] User devices: devices such as smartphones, tablets, and PCs

[2699] Server: A computer system responsible for data processing and generation.

[2700] Database: A storage system that stores user information, alarm settings, request information, etc.

[2701] AI engines: natural language processing engine, deepfake generation engine, speech synthesis engine, emotion engine

[2702] User Authentication

[2703] The user launches the app and enters their login information. The device encrypts the input information and sends it to the server, which then queries the database to obtain the authentication result. If authentication is successful, the home screen is displayed; if it is unsuccessful, an error message is displayed on the device. This uses data encryption using AES (Advanced Encryption Standard) and authentication using OAuth tokens.

[2704] Conversation function using emotion recognition

[2705] The user selects the conversation function and inputs a message by voice or text. The device recognizes the voice input and converts it into text. This process uses the Google Cloud Speech-to-Text API. The converted text data is sent to the server, where a natural language processing engine (e.g., OpenAI's GPT-3) generates an appropriate response. The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotions and reflect them in the response. Finally, the server uses a deepfake generation engine (e.g., DeepFaceLab) and a speech synthesis engine (e.g., Amazon Polly) to generate video and audio, which are then sent to the device for playback.

[2706] Alarm setting using emotion recognition

[2707] The user opens the alarm settings screen and enters the desired alarm time and wake-up message. The device sends the setting information to the server, which stores the data in a database. When the alarm time comes, the server retrieves the corresponding alarm data and uses an emotion engine to generate deepfake video and audio that matches the user's emotions. The generated data is sent to the device, which then plays the video and audio along with the alarm. DeepFaceLab and Amazon Polly are used in this process.

[2708] Song Request Using Emotion Recognition

[2709] The user inputs a song request, and the device sends the request information to the server. The server then checks the database for copyright information. If the copyright is permitted, the server uses an emotion engine to recognize the user's emotions and generate deepfake video and audio. This uses DeepFaceLab and Amazon Polly. Finally, the generated data is sent to the device, where the user can enjoy the video and audio.

[2710] Specific examples

[2711] Example 1: Conversation function

[2712] When a user launches the app and logs in, the home screen appears, and they tap the conversation function icon to say, "Hello, how are you today?" The device converts the speech into text and sends it to the server. The server analyzes the text and generates a response such as, "I'm feeling good today, how about you?" The emotion engine recognizes emotions from the user's voice and reflects this in the response. The deepfake generation engine generates video that matches the user's emotions, and the speech synthesis engine adjusts the tone to generate the audio. The device plays back the video and audio, and the "oshi" character says, "I'm feeling good today, how about you?"

[2713] Example 2: Alarm settings

[2714] The user opens the alarm settings and sets it to "Good morning, it's time to wake up" at 7:00. When the alarm time comes, the server receives the message "Good morning, it's time to wake up" and uses the emotion engine to generate video and audio that matches the user's emotion. While the device sounds the alarm, it plays video and audio that matches the emotion, and the "oshi" wakes the user up.

[2715] Prompt Sentence Examples

[2716] "Simulate a scenario where a user uses emotion-aware conversation features."

[2717] "Please explain the steps a user goes through to set up an alarm using deepfake technology."

[2718] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2719] User Authentication

[2720] Step 1:

[2721] A user launches the app and accesses the login screen, where they enter their use...

Claims

1. In a data processing device, means for receiving authentication information entered by a user; means for querying a database for said authentication information and obtaining an authentication result; means for displaying a home screen or an error message for the user based on the authentication result; means for recognizing voice and converting input data from a user into text data; natural language processing means for analyzing the text data and generating an appropriate response; means for generating deepfake video and audio data based on the responses; means for transmitting the generated video and audio data to a user terminal and playing the data; A system including:

2. means for receiving an alarm time and message set by a user; A means for generating deepfake video and audio data based on the alarm time; means for reproducing the generated data on a terminal at an alarm time; The system of claim 1 further comprising:

3. means for receiving user-entered song request information; means for verifying copyright information for the requested song; A means for generating deepfake video and audio data, where copyright permits; and means for transmitting the generated data to a user terminal and playing it back; The system of claim 1 further comprising:

Images