system

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting visual information; means for collecting audio information; means for collecting textual information; means for pre-processing the information; means for training a generative model using the pre-processed information; means for generating visual and audio information using the trained generative model; and means for storing the generated information as a NFT.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The market value of celebrities often declines or is lost due to aging or death. As a result, the celebrity's value in a company's advertising strategy also declines, negatively impacting the brand image. Furthermore, if a company wants to maintain a celebrity's popularity after their prime has passed, their only option is to reuse past footage and audio. However, these methods are likely to create visual and audio incongruities, making them hardly an effective advertising strategy. This invention aims to solve these problems and realize an effective advertising strategy while perpetually maintaining the celebrity's market value. [Means for solving the problem]

[0005] This invention relates to a system that collects visual, audio, and text information, preprocesses it, and then trains a generative model. Specifically, the collected visual information is used to extract a talent's facial features, the audio information is used to identify the talent's vocal characteristics, and the text information is used to analyze the talent's thought patterns. This preprocessed data is then used to train a generative model (e.g., a GAN or Transformer model) to generate video and audio that faithfully reproduces the talent's appearance in their prime. The visual and audio information obtained during this generation process can be stored as NFTs, which can retain their value for a long period of time. This system allows companies to develop high-quality, consistent advertising strategies that persist even after a talent's prime has passed.

[0006] "Visual information" refers to information in a visible form, such as images or videos of celebrities.

[0007] "Audio information" refers to information related to the auditory sense, such as the voice of a celebrity or music.

[0008] "Text information" refers to information recorded in written form, such as the contents of a celebrity's interviews, social media posts, and various past statements.

[0009] "Preprocessing" refers to the preparation of collected data by standardizing, removing noise, tokenizing, etc., to make it into a format that can be processed by a generative model.

[0010] A "generative model" is a machine learning model that generates new visual and audio information based on collected data.

[0011] "NFT" is an abbreviation for Non-Fungible Token, a technology for recording the uniqueness and ownership of digital assets on the blockchain.

[0012] "User" refers to an individual or company that uses this system to recreate the heyday of a celebrity and use it for advertising strategies, etc.

[0013] The "server" is a high-performance computer that processes data for the entire system and trains generative models.

[0014] A "terminal" is a device (e.g., a PC or smartphone) that a user uses to send data to a server and receive generated results. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The system of the present invention collects visual, audio, and textual information, preprocesses the data, and then trains a generative model to generate visual and audio information that recreates the talent's appearance in their prime. The generated information is stored as an NFT, which can retain its value for a long period of time.

[0037] server

[0038] Data collection

[0039] The server receives visual, audio, and textual information uploaded by users, including talent images, video files, audio files, and social media posts.

[0040] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[0041] Data Preprocessing

[0042] The server analyzes the facial features of the received visual information, denoises the audio information, and tokenizes the text information.

[0043] Example: A server extracts facial features from a movie clip, removes noise from an audio file, or tokenizes and extracts features from a social media post.

[0044] Training a generative model

[0045] The server uses the preprocessed data to train a generative model (e.g., GAN or Transformer), which is used to integrate and reproduce the talent's visual, vocal, and thought patterns.

[0046] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[0047] AI Talent Generation

[0048] The server uses the trained generative model to generate visual and audio information for the talent based on the specified scenario.

[0049] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[0050] NFT conversion

[0051] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[0052] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[0053] Terminal

[0054] Entering and Submitting Data

[0055] The terminal checks for errors in the visual, audio, and text information entered by the user and sends the information to the server.

[0056] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[0057] Receiving and displaying the generated results

[0058] The terminal receives the generated visual and audio information from the server and displays it to the user.

[0059] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[0060] User

[0061] Talent selection and data provision

[0062] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[0063] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[0064] Check the generated results and request corrections

[0065] The user reviews the generated talent's visual and audio information and requests modifications from the server if necessary.

[0066] Example: A marketing person reviews the generated video and requests the server to correct specific facial expressions or lines.

[0067] final use

[0068] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[0069] Example: A marketer distributes the final video for an advertising campaign.

[0070] The system of the present invention makes it possible to execute a high-quality and consistent advertising strategy while perpetually maintaining the market value of a talent even after the talent has passed its prime.

[0071] The processing flow will be explained below.

[0072] Program processing flow

[0073] server

[0074] Step 1:

[0075] The server receives visual, audio, and textual information uploaded by users, including celebrity images, video files, audio files, and social media posts.

[0076] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[0077] Step 2:

[0078] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[0079] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[0080] Step 3:

[0081] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[0082] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[0083] Step 4:

[0084] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[0085] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[0086] Step 5:

[0087] The server uses the preprocessed data to train a generative model, integrating visual, audio, and text information to build a generative model that can recreate the talent in their prime.

[0088] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[0089] Step 6:

[0090] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[0091] Example: A server generates video and audio of an actor introducing a new product based on a specific scenario.

[0092] Step 7:

[0093] The server creates a digital signature and records it on the blockchain to store the generated visual and audio information as an NFT.

[0094] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[0095] Terminal

[0096] Step 1:

[0097] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[0098] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[0099] Step 2:

[0100] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[0101] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[0102] User

[0103] Step 1:

[0104] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[0105] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[0106] Step 2:

[0107] The user can check the generated visual and audio information of the talent and request corrections from the server as necessary, thereby creating optimal advertising materials.

[0108] Example: A marketing person reviews the generated video and requests the server to correct specific facial expressions or lines.

[0109] Step 3:

[0110] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[0111] Example: A marketer distributes the final video for an advertising campaign.

[0112] The above is a specific processing flow of the present invention, which makes it possible to execute a high-quality and consistent advertising strategy while perpetually maintaining the market value of a talent even after they have passed their prime.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] Conventional digital talent reproduction technologies rely on a single source of information (such as visual information only or audio information only), making it difficult to reproduce a talent's peak performance. Furthermore, there is no way to preserve the value of the generated digital data over the long term. This makes it difficult to implement high-quality, consistent advertising strategies.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative AI model using the preprocessed information, means for generating visual and audio information using the trained generative AI model, and means for digitally signing the generated visual and audio information and recording it on a blockchain as a non-fungible token. This makes it possible to reproduce a talent's peak years with high accuracy and preserve the value of the generated digital data for a long period of time.

[0118] "Visual information" refers to information that can be seen and recognized with the eyes, such as images and videos.

[0119] "Audio information" refers to information that can be heard by the ears, such as audio data or recording files.

[0120] "Text information" refers to information such as sentences and character data, the contents of which can be understood by reading it.

[0121] "Preprocessing" refers to a series of processes that convert collected information into a form suitable for input into a generative model, such as parsing, formatting, and tokenization.

[0122] "Generative AI model" refers to an artificial intelligence algorithm that is trained using collected and preprocessed data to generate new data (visual and audio information) based on specific inputs.

[0123] A "non-fungible token (NFT)" refers to a unique digital token that can prove ownership of a digital asset using blockchain technology.

[0124] "Digital Signature" refers to a cryptographic signature created to prove that digital data has not been tampered with.

[0125] "Blockchain" refers to a technology that maintains a series of records of digital transactions and manages them in a distributed manner across multiple computers.

[0126] The system of the present invention collects visual, audio, and text information, preprocesses the data, and then trains a generative AI model to generate visual and audio information that recreates the talent's appearance in their prime. The generated information is stored as a non-fungible token (NFT), which can retain its value for a long period of time.

[0127] server

[0128] The server uses HTTP requests to receive visual, audio, and textual information uploaded by users, such as images, video files, audio files, social media posts, etc. The server then validates the data and stores it in a database.

[0129] When preprocessing visual information, we use libraries such as OpenCV and Dlib to analyze facial features. When preprocessing audio information, we use audio processing libraries such as LibROSA to remove noise. When preprocessing text information, we use natural language processing libraries such as SpaCy and NLTK to tokenize and extract features.

[0130] TensorFlow and PyTorch are used to train generative AI models (e.g., GANs and Transformers) using the preprocessed data, which then builds highly accurate models that integrate the talent's appearance, voice, and thought patterns.

[0131] Based on the trained generative AI model, the server generates visual and audio information according to prompts given by the client. For example, a prompt such as "Generate a video in which actor X from the 1990s introduces new product Y. Product Y's features are A and B, and actor X should introduce the product in an enthusiastic and friendly tone."

[0132] The generated visual and audio information is digitally signed and recorded on the blockchain as a non-fungible token (NFT), using blockchain technologies such as Ethereum and Flow.

[0133] Terminal

[0134] The device checks the visual, audio, and text information entered by the user for errors and sends it to the server in the appropriate format, for example, checking the format of a movie clip, checking the sampling rate of an interview audio, and ensuring that a social media post is properly formatted.

[0135] It receives generated visual and audio information from the server, analyzes the generated results, and displays them on the screen or plays them back to the user. The user can review the generated content and request corrections if necessary.

[0136] User

[0137] Users log in to the system, select a period of time they want to focus on for a talent, and upload visual, audio, and text information related to that period. For example, a marketer could select an actor's data from the 1990s and upload film clips, audio interviews, and social media posts from that period.

[0138] Users can check the visual and audio information of the generated talent and request the server to modify specific facial expressions, lines, etc. Finally, they can select the content they are satisfied with and use it in corporate advertising campaigns and promotions, enabling them to execute high-quality, consistent advertising strategies while accurately recreating the talent in their prime.

[0139] This system allows users to easily create, store, and use digital information about celebrities, even after they have passed their prime, while maintaining their market value in perpetuity.

[0140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0141] Step 1: Enter and submit data

[0142] Subject: Terminal

[0143] Specific operation: The user inputs visual, audio, and text information through the terminal. The terminal checks for errors in this data and converts it into the required format. Specifically, it checks the format of the movie clip and the audio sampling rate.

[0144] Input: User-uploaded film clips, interview audio files, social media posts

[0145] Output: Error checked and formatted data

[0146] Data processing: The device checks the format and data integrity and sends it to the server via an HTTP request.

[0147] Step 2: Receiving and storing data

[0148] Subject: Server

[0149] What happens: The server receives the HTTP request, checks the file format, verifies that it is in the correct format, and saves it to the database.

[0150] Input: Error-checked and formatted data sent from the terminal

[0151] Output: Visual, audio and text information stored in a database

[0152] Data processing: The server checks the data format and registers it in the database in a consistent state.

[0153] Step 3: Preprocessing the data

[0154] Subject: Server

[0155] Specific operation: The server uses libraries such as OpenCV and Dlib to extract facial features from visual information, audio processing libraries such as LibROSA to remove noise from audio data, and text information is tokenized and features are extracted using SpaCy and NLTK.

[0156] Input: Visual, audio, and text information stored in a database

[0157] Output: Preprocessed visual, audio, and text information

[0158] Data processing: The server analyzes visual information, reduces audio noise, tokenizes text, and extracts features.

[0159] Step 4: Training the generative AI model

[0160] Subject: Server

[0161] How it works: The server uses the preprocessed data to train a generative AI model (such as a GAN or Transformer). The training process is performed using TensorFlow or PyTorch to improve the model's accuracy.

[0162] Input: Preprocessed visual, audio, and text information

[0163] Output: A trained generative AI model

[0164] Data Computation: The server processes the data in batches and runs a training loop to improve the model's performance. It monitors the model's accuracy and loss function and adjusts hyperparameters as needed.

[0165] Step 5: Generate AI talent

[0166] Subject: Server

[0167] How it works: The server uses a trained generative AI model to generate visual and audio information based on prompts provided by the client.

[0168] Input: prompt, trained generative AI model

[0169] Output: Generated visual and audio information

[0170] Data calculation: The server analyzes the given prompt and inputs it into the generative AI model, which generates visual and audio information and retrieves that data.

[0171] Step 6: NFT

[0172] Subject: Server

[0173] How it works: The server digitally signs the generated visual and audio information and records it on the blockchain as a non-fungible token (NFT). It uses blockchain technologies such as Ethereum and Flow.

[0174] Input: Generated visual and audio information

[0175] Output: NFT recorded on the blockchain

[0176] Data processing: The server creates a digital signature and records ownership information on the blockchain.

[0177] Step 7: Receive and display the generated results

[0178] Subject: Terminal

[0179] Specific operation: The device receives the visual and audio information generated by the server and displays it to the user. The generated results are analyzed and displayed on the screen or played back as audio.

[0180] Input: Generated visual and audio information sent from the server

[0181] Output: Visual and audio information displayed to the user

[0182] Data processing: The device analyzes the generated data and displays and plays it appropriately for the user.

[0183] Step 8: Check the results and request corrections

[0184] Subject: User

[0185] Specific operation: The user checks the generated visual and audio information and requests the server to correct specific facial expressions, lines, etc.

[0186] Input: Generated visual and audio information

[0187] Output: Correction request

[0188] Data processing: The user checks the generated results, clarifies any corrections, and sends a request to the server.

[0189] Through these processing steps, users can create high-quality digital representations of their talent and use the generated information in advertising campaigns and promotions.

[0190] (Application example 1)

[0191] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0192] Conventional methods have had the problem that once a celebrity's prime passes, their market value declines, limiting their use in advertising and promotions. Furthermore, it is difficult to generate high-quality visual and audio information, and there is no way to maintain its value over the long term. Furthermore, when applied to virtual stores, there is a lack of means to enable real-time interactive customer service.

[0193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0194] In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for recording the generated information on a blockchain and creating a digital signature for maintaining its value over the long term, and means for reproducing the generated visual and audio information in a virtual environment based on a user-specified scenario and displaying it to the user. This makes it possible to provide interactive customer service in a virtual store while perpetually maintaining the market value of a talent even after they have passed their prime.

[0195] "Visual information" refers to images, video clips, and other visual data.

[0196] "Audio information" refers to sound files, recordings, and other audio data.

[0197] "Text information" refers to sentences, notes, text files, and other text data.

[0198] "Preprocessing" refers to the process of analyzing collected data, removing noise, and extracting features.

[0199] "Generative model" refers to a trained algorithm for generating visual and audio information using artificial intelligence techniques.

[0200] "Digital signature" refers to a cryptographic technique used to ensure the authenticity and authenticity of generated information.

[0201] "Blockchain" refers to a system that links information in a chain and uses distributed ledger technology to prevent data tampering.

[0202] A "virtual environment" refers to a virtual space or situation generated by computer simulation.

[0203] This invention is a system for realizing interactive customer service in a virtual environment, and is composed of three elements: a server, a terminal, and a user.

[0204] server

[0205] The server collects visual, audio, and text information and preprocesses it. Specifically, the following processes are performed:

[0206] 1. Data Collection

[0207] The server receives visual information (e.g., images, video clips), audio information (e.g., audio recordings), and text information (e.g., social media posts) uploaded by users.

[0208] 2. Data Preprocessing

[0209] The received visual information is analyzed for facial features using an image processing library (e.g., OpenCV).

[0210] For audio information, an audio processing library is used to remove noise.

[0211] For text information, tokenization is performed using a natural language processing library (e.g., Transformers).

[0212] 3. Training the generative model

[0213] The pre-processed data is used to train a generative AI model (e.g., GAN or Transformer) that integrates the talent's visual, vocal, and thought patterns.

[0214] 4. AI Talent Generation

[0215] A trained generative model is used to generate visual and audio information based on a user-specified scenario.

[0216] 5. NFT conversion

[0217] The generated visual and audio information is recorded on the blockchain, a digital signature is created and stored as an NFT.

[0218] Terminal

[0219] The terminal has the following features:

[0220] 1. Enter and submit data

[0221] The visual, audio, and text information entered by the user is checked for errors and then sent to the server.

[0222] 2. Receiving and displaying the generated results

[0223] The system receives visual and audio information generated by the server and displays it to the user. Specifically, it plays video and audio of an AI talent introducing products in a virtual store environment.

[0224] User

[0225] The user performs the following operations:

[0226] 1. Talent selection and data provision

[0227] Users log in, select a talent's prime period, and upload visual, audio, and text information related to that period.

[0228] 2. Check the generated results and request corrections

[0229] Check the visual and audio information of the generated AI talent and request corrections from the server if necessary.

[0230] 3. End use

[0231] Finally, the content that satisfies the customer is selected and used for interactive customer service in the virtual store.

[0232] Specific examples

[0233] For example, a marketer could collect data on popular actors from the 1990s and generate a scenario to introduce a new product in a virtual store.

[0234] "Generate a scenario in which a popular actor from the 1990s appears in a virtual store and gives a detailed explanation of new products."

[0235] This system enables interactive customer service in a virtual store while perpetually maintaining the market value of talent even after they have passed their prime. The hardware used is a server and end-user devices, and the specific software includes OpenCV, Transformers, audio processing libraries, blockchain technology, etc.

[0236] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0237] Step 1:

[0238] The server collects visual, audio, and text information, including images, video clips, audio recordings, and social media posts uploaded by users. Specific data formats used include JPEG, MP4, MP3, and TXT. The received data is then temporarily stored in the server's database.

[0239] Step 2:

[0240] The server preprocesses the collected visual, audio, and text information. Specifically, for visual information, OpenCV is used to analyze facial features and a facial recognition algorithm is used to extract facial landmarks (eyes, nose, mouth, etc.). For audio information, an audio processing library is used to remove noise and convert it into clear audio data. For text information, the Transformers library is used to tokenize the data and convert each word or phrase into a vector representation. This preprocessed data is compiled into a training dataset and used in the next stage.

[0241] Step 3:

[0242] The server uses the preprocessed data to train a generative model. Specifically, the training dataset is input into a GAN (generative adversarial network) or Transformer model, and the model parameters are optimized. This results in a generative AI model capable of integrating and reproducing the talent's visual, vocal, and thought patterns. Model training is performed efficiently using a GPU to increase computational speed.

[0243] Step 4:

[0244] The server generates visual and audio information using a trained generative model. Based on a scenario specified by the user, a prompt sentence, such as "A celebrity will introduce a new product in a virtual store," is input into the generative model. Based on this prompt sentence, the model generates the celebrity's visual information (e.g., facial and body movements) and audio information (e.g., audio explaining the product). The generated data is temporarily stored in the server's storage.

[0245] Step 5:

[0246] The server stores the generated visual and audio information as an NFT. Specifically, it creates a digital signature for the generated data and records the information using blockchain technology. This guarantees the uniqueness of the generated data, making it possible to maintain its reliability and authenticity over the long term.

[0247] Step 6:

[0248] The device receives and displays the generated results. Specifically, it receives the generated visual and audio information sent from the server and displays and plays it for the user. For example, it plays video and audio of an AI talent introducing a product in a simulated virtual store environment. Based on this, the user can understand the details of the product and consider purchasing it.

[0249] Step 7:

[0250] The user checks the generated results and requests corrections. Specifically, the user requests the server to make minor corrections to the generated video and audio (e.g., changing specific facial expressions or lines). The server receives this request, makes corrections using the generative model again, and provides a satisfactory result.

[0251] Step 8:

[0252] Users can then select the content they are most satisfied with and use it in the virtual store. The generated AI talent's video and audio can then be used in actual promotions and advertising campaigns. This provides users with an interactive customer service experience and effectively communicates the appeal of the products.

[0253] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0254] The system of the present invention collects visual, audio, and text information, preprocesses the data, and then trains a generative model to generate visual and audio information that recreates the celebrity's appearance in their prime. Furthermore, by incorporating an emotion engine that recognizes user emotions and incorporating user feedback into the generation process, more optimal content generation is achieved. Furthermore, the generated information is stored as an NFT, which can maintain its value for a long period of time.

[0255] server

[0256] Data collection

[0257] The server receives visual, audio, and textual information uploaded by users, including talent images, video files, audio files, and social media posts.

[0258] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[0259] Data Preprocessing

[0260] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[0261] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[0262] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[0263] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[0264] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[0265] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[0266] Training a generative model

[0267] The server uses the preprocessed data to train a generative model (e.g., GAN or Transformer), which is used to integrate and reproduce the talent's visual, vocal, and thought patterns.

[0268] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[0269] AI Talent Generation

[0270] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[0271] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[0272] Utilizing the Emotion Engine

[0273] The server adjusts the output of the generative model using an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's feedback and optimizes the generated visual and audio information based on the results.

[0274] Example: The server recognizes the user's emotions in real time and incorporates that feedback into the generative model.

[0275] NFT conversion

[0276] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[0277] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[0278] Terminal

[0279] Entering and Submitting Data

[0280] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[0281] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[0282] Receiving and displaying the generated results

[0283] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[0284] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[0285] Feedback collection with emotion engine

[0286] The terminal transmits the emotions of the user while viewing the generated content to the emotion engine in real time.

[0287] Example: The device analyzes the user's facial expressions and speech while watching, and sends the emotional data to the server's emotion engine.

[0288] User

[0289] Talent selection and data provision

[0290] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[0291] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[0292] Check the generated results and request corrections

[0293] The user can review the visual and audio information of the generated talent, request corrections from the server if necessary, and provide feedback using the emotion engine to optimize the generation process.

[0294] Example: A marketer reviews the generated video and requests the server to modify specific facial expressions or lines, and the emotion engine learns from user feedback.

[0295] final use

[0296] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[0297] Example: A marketer distributes the final video for an advertising campaign.

[0298] The system of the present invention makes it possible to execute a high-quality, consistent advertising strategy that is in line with user emotions while perpetually maintaining the market value of a talent even after the talent has passed its prime.

[0299] The processing flow will be explained below.

[0300] Program processing flow

[0301] server

[0302] Step 1:

[0303] The server receives visual, audio, and textual information uploaded by users, including celebrity images, video files, audio files, and social media posts.

[0304] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[0305] Step 2:

[0306] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[0307] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[0308] Step 3:

[0309] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[0310] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[0311] Step 4:

[0312] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[0313] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[0314] Step 5:

[0315] The server uses the preprocessed data to train a generative model, integrating visual, audio, and text information to build a generative model that can recreate the talent in their prime.

[0316] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[0317] Step 6:

[0318] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[0319] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[0320] Step 7:

[0321] The server uses an emotion engine to analyze the user's emotions and reflects them as feedback to the generative model, thereby improving the quality of the generated content.

[0322] Example: The server collects data on the user's emotions and reflects it in the generation process.

[0323] Step 8:

[0324] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[0325] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[0326] Terminal

[0327] Step 1:

[0328] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[0329] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[0330] Step 2:

[0331] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[0332] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[0333] Step 3:

[0334] The device transmits emotional data in real time to a server while the user is viewing the generated content, and emotion recognition software is used to analyze emotions such as facial expressions and speech.

[0335] Example: The device analyzes the user's facial expressions and speech while watching and sends the emotional data to the server.

[0336] User

[0337] Step 1:

[0338] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[0339] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[0340] Step 2:

[0341] The user can review the visual and audio information of the generated talent, request corrections from the server if necessary, and provide feedback using the emotion engine to optimize the generation process.

[0342] Example: A marketer reviews the generated video and requests the server to modify specific facial expressions or lines, and the emotion engine learns from user feedback.

[0343] Step 3:

[0344] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[0345] Example: A marketer distributes the final video for an advertising campaign.

[0346] The above is a specific processing flow of the present invention. This system makes it possible to execute a high-quality, consistent advertising strategy that is in tune with the user's emotions while perpetually maintaining the market value of a talent even after they have passed their prime.

[0347] Example 2

[0348] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0349] In today's digital media environment, there is a need to generate high-quality, consistent visual and audio content while reusing the assets of past talent. However, current technologies lack a method for consistently processing visual, audio, and text information and training effective generative models, resulting in inconsistent quality of generated content. Furthermore, the lack of a function for adjusting content based on user emotions makes it difficult to provide an optimal entertainment experience.

[0350] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for extracting facial features from the visual information of the preprocessed information, means for denoising the preprocessed audio information, means for tokenizing and analyzing the preprocessed text information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for adjusting the generated visual and audio information based on user emotional feedback, and means for storing the generated information as an NFT. This enables effective reuse of past talent assets and the generation of high-quality, consistent visual and audio content adjusted based on user emotions.

[0351] "Visual information" is information that is presented in a form that can be seen with the eyes, such as an image or video file.

[0352] "Audio information" refers to information that is expressed in a form that can be heard through the ears, such as audio files or recorded data.

[0353] "Text information" refers to information that is written in a readable form, such as written characters or sentences.

[0354] "Preprocessing" refers to the process of preparing collected raw data in a form that is easier to analyze, and includes processes such as noise removal and tokenization.

[0355] "Facial features" refers to information extracted from visual information about a person's facial shape and characteristic points such as the eyes, nose, and mouth.

[0356] "Noise reduction" is a process for removing unnecessary background sounds and interference sounds contained in audio information.

[0357] "Tokenization" is a process of dividing text information into small units such as words and phrases and analyzing their meanings and relationships.

[0358] A "generative model" is an algorithm used to generate new visual or audio information based on input data, and includes GANs and Transformers.

[0359] A "trained generative model" is a generative algorithm that has been optimized using preprocessed data and is used to generate new content based on a specified scenario.

[0360] "Emotional feedback" is a process of collecting evaluations and opinions based on the user's emotional state and adjusting the generated content based on that information.

[0361] An "NFT" is a token that uses blockchain technology to prove ownership of digital data and represents a one-of-a-kind digital asset.

[0362] This system generates high-quality visual and audio information by integrating visual, audio, and text information, preprocessing the data, and training a generative model. The generated information is stored as NFTs, which retain their value over the long term. This system provides optimized content based on specific scenarios and user emotional feedback.

[0363] server

[0364] Data collection

[0365] The server receives visual, audio, and text information uploaded by the user through the device. Specifically, it receives HTTP requests and saves image files and text data in storage.

[0366] Example: A server receives and stores actor film clips, interview audio files, and social media posts sent by a marketer.

[0367] Data Preprocessing

[0368] The server preprocesses the collected visual information and extracts facial features using a facial recognition algorithm such as FaceNet. It also preprocesses the audio information by removing noise using FFmpeg or Audacity. It then tokenizes and analyzes the text information using natural language processing libraries such as NLTK and spaCy.

[0369] Examples: Extracting actors' facial features from movie clips, removing background noise from interview audio, and tokenizing and analyzing social media posts.

[0370] Training a generative model

[0371] The server uses the preprocessed data to train a generative model such as a GAN or Transformer, using TensorFlow or PyTorch to integrate facial features, audio data, and text data to optimize the model.

[0372] Example: Training a generative model of AI talent using facial features, voice features, and thought patterns derived from text data.

[0373] AI Talent Generation

[0374] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a given scenario, inputs a prompt (e.g., "Introducing a new product") into the model, and stores the generated video and audio.

[0375] Example: Based on a specific product promotion scenario, generate video and audio of an actor introducing a product.

[0376] Utilizing the Emotion Engine

[0377] The server uses an emotion engine to analyze user feedback and adjust the generated content. The emotion engine recognizes user emotions in real time and fine-tunes the generative model based on them.

[0378] Example: Collecting the user's emotions as feedback and adjusting the actor's facial expressions based on that data.

[0379] NFT conversion

[0380] The server stores the generated talent's visual and audio information as an NFT, which is recorded on the Ethereum blockchain with a digital signature, making it a unique digital asset.

[0381] Example: The generated video and audio of an actor is registered as an NFT, and its ownership information is recorded on the blockchain.

[0382] Terminal

[0383] Entering and Submitting Data

[0384] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits it to the server.

[0385] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[0386] Receiving and displaying the generated results

[0387] The terminal receives the generated content from the server and displays it to the user, allowing the user to review the results and make correction requests if necessary.

[0388] Example: Display a video of an actor introducing a new product sent from the server and play an audio file.

[0389] Feedback collection with emotion engine

[0390] While the user is viewing the generated content, the device analyzes facial expressions and speech in real time and sends the results to the emotion engine.

[0391] Example: Analyze the facial expressions and speech of the user while watching, and send the emotional data to the emotion engine on the server.

[0392] User

[0393] Talent selection and data provision

[0394] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[0395] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[0396] Check the generated results and request corrections

[0397] The user can review the generated visual and audio information, request corrections from the server, and even provide feedback using the emotion engine to optimize the generation process.

[0398] Example: The generated video is reviewed, and specific facial expressions or lines are corrected by the server, and the emotion engine learns from the user's feedback.

[0399] final use

[0400] The user ultimately selects the content they are satisfied with and uses it for advertising campaigns and promotions.

[0401] Example: A marketer distributes the final video for an advertising campaign.

[0402] Prompt Sentence Examples

[0403] "Generate a new product promotional video using the face and voice of a famous actor from the 1980s. Use uploaded film clips and interview audio."

[0404] "Please use the user's emotional data as a reference to optimize the facial expressions of the actors in the generated video to make them appear happier."

[0405] The above is an embodiment of the present invention.

[0406] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0407] Step 1: Data collection

[0408] The server receives visual, audio, and text information uploaded by users through their devices. Specifically, it receives HTTP requests and saves image files and text data in storage. The input to this process is the data provided by the user, and the output is the saved raw data. Specifically, the server receives and saves actor film clips, interview audio files, and social media posts sent by marketers.

[0409] Step 2: Data preprocessing (visual information)

[0410] The server preprocesses the received visual information and extracts facial features using a facial recognition algorithm such as FaceNet. The input to this process is the collected visual information, and the output is the extracted facial feature information. Specifically, the server uses OpenCV to analyze frames from a movie clip, detect facial regions, and extract feature points (eyes, nose, mouth, etc.).

[0411] Step 3: Data preprocessing (audio information)

[0412] The server uses FFmpeg and Audacity to preprocess the audio information and remove noise. The input of this process is the collected audio information, and the output is clear audio data with noise removed. Specifically, the server uses FFmpeg to read the audio file, apply a filter to remove noise, and generate clear audio data.

[0413] Step 4: Data preprocessing (text information)

[0414] The server preprocesses the text information and performs tokenization using a natural language processing library (NLTK or spaCy). The input to this process is the collected text information, and the output is tokenized text data. Specifically, the server reads the text data, splits it into tokens, extracts keywords and important phrases, and stores them in a database.

[0415] Step 5: Training the generative model

[0416] The server uses the preprocessed data to train a generative model such as a GAN or Transformer. The input for this process is the preprocessed visual, audio, and text information, and the output is a trained generative model. Specifically, the server uses TensorFlow or PyTorch to train the model using facial features, audio data, and text data as input, and saves the trained model.

[0417] Step 6: Generate AI talent

[0418] The server uses a trained generative model to generate visual and audio information for an AI talent based on a specified scenario. The input to this process is the scenario information and the trained generative model, and the output is the generated visual and audio information. Specifically, the server inputs a prompt sentence (e.g., "Introducing a new product") into the model and saves the generated video and audio.

[0419] Step 7: Leverage your emotional engine

[0420] The device analyzes the user's facial expressions and speech in real time while watching the generated content and sends the results to the emotion engine. The input for this process is the user's facial expression data and voice data, and the output is emotional feedback. Specifically, the device's camera and microphone are used to collect the user's facial expression data and voice data, and the data is sent to the emotion engine in real time.

[0421] The server uses an emotion engine to analyze user feedback and adjusts the generated content by incorporating the results into the generative model. The input to this process is the emotional feedback sent from the device, and the output is the adjusted generated content. Specifically, the emotion engine identifies the user's emotions and optimizes the generated video and audio based on them.

[0422] Step 8: NFT

[0423] The server stores the generated visual and audio information of the talent as an NFT. The input of this process is the generated visual and audio information, and the output is an NFT recorded on the blockchain. Specifically, the server digitally signs the generated data and records it on a blockchain network such as Ethereum.

[0424] (Application example 2)

[0425] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0426] In the prior art, there was no effective method for generating visual and audio information to recreate a specific celebrity's peak performance, and it was difficult to reflect user emotional feedback in real time. Furthermore, there was insufficient protection of the value of the generated content and proof of ownership, making it difficult to continue providing high-quality content that satisfies users. The present invention aims to solve these problems.

[0427] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for storing the generated information as a non-fungible token, and means for collecting and analyzing user feedback using an emotion engine to allow users to customize the generated visual and audio information and reflecting the feedback in the generation process. This makes it possible to continue providing high-quality, customizable content and also makes it easier to preserve the value of the generated content and prove ownership.

[0428] "Visual information" is a general term for image and video data collected from users and other sources.

[0429] "Audio Information" refers collectively to audio data collected from users and other sources.

[0430] "Text Information" refers collectively to written or typed character information collected from users or other sources.

[0431] "Preprocessing" is the process of formatting collected visual, audio, and text information to make it easier to analyze.

[0432] A "generative model" is a machine learning or deep learning algorithm that uses collected and preprocessed data to generate new visual or audio information.

[0433] A "non-fungible token" is a digital asset with unique value based on a blockchain, typically used for proof of ownership and store of value purposes.

[0434] An "emotion engine" is an algorithm and system that analyzes users' emotions in real time and collects and reflects the results as feedback.

[0435] "User feedback" refers to the user's impressions and reactions to the generated visual and audio information, and requests for modifications based on the impressions and reactions.

[0436] This invention provides a system that collects visual, audio, and text information, preprocesses them, and then generates visual and audio information using a generative model. The system includes a means for storing the generated information as a non-fungible token (NFT). The system also uses an emotion engine to allow users to customize the generated visual and audio information, collects and analyzes user feedback, and incorporates that feedback into the generation process.

[0437] System configuration and operation overview

[0438] 1. Visual information collection and preprocessing:

[0439] The server receives image and video files uploaded by users, such as old movie clips or interview footage.

[0440] Use OpenCV to resize and reshape the image and extract facial features.

[0441] 2. Acquisition and preprocessing of audio information:

[0442] The server receives audio files uploaded by users, for example, interview audio data.

[0443] Use pydub to convert the audio data to mono and format it in a way that is suitable for the speech recognition model.

[0444] 3. Text information collection and preprocessing:

[0445] The server receives text information uploaded by the user, such as a social media post or a script.

[0446] Text information is tokenized and converted into a format that can be input into natural language processing models such as GPT-2.

[0447] 4. Training the generative model and generating visual and audio information:

[0448] The server trains a generative model (e.g., GAN, Transformer) using the preprocessed data.

[0449] A trained generative model is used to generate visual and audio information.

[0450] 5. Feedback collection and analysis using emotion engine:

[0451] The terminal presents the generated visual and audio information to the user and collects the user's emotional feedback in real time.

[0452] The emotion engine analyzes the user's facial expressions and voice and sends the feedback to the server.

[0453] 6. Optimization of generated information and NFTization:

[0454] The server optimizes the generation process based on user feedback.

[0455] The final content generated is stored as a non-fungible token (NFT) to preserve value and prove ownership.

[0456] Specific examples

[0457] If an advertising agency wants to use a famous celebrity from the 1990s to promote a new product, they can use this system. They can upload old movie clips, interview audio, social media posts, etc. to train a generative model. The system then generates an advertising video, and users can provide real-time feedback via smartphone or head-mounted display. The final optimized advertising video is stored as an NFT, preserving ownership and value.

[0458] Example prompt sentence:

[0459] "Generate an advertising video for new product X featuring famous 90s actor A. Use the following information: movie clips, interview audio, and social media posts."

[0460] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0461] Step 1:

[0462] The server collects visual information from users. Specifically, users upload image or video files, such as old movie clips or interview footage. The server receives these files, resizes and reshapes the images using OpenCV, and extracts facial features.

[0463] Input: User-uploaded film clips and interview footage

[0464] Processing: Resize, shape, and extract facial features using OpenCV

[0465] Output: Preprocessed visual data

[0466] Step 2:

[0467] The server collects audio information from users, specifically, by uploading interview recordings or other audio files. The server receives these audio files and uses pydub to convert them to mono and remove background noise.

[0468] Input: Audio file uploaded by the user

[0469] Processing: Mono conversion and noise removal using pydub

[0470] Output: Preprocessed audio data

[0471] Step 3:

[0472] The server collects text information from users. Specifically, users upload text data such as social media posts and scripts. The server receives this text information, tokenizes it, and converts it into a format suitable for input into a natural language processing model (such as GPT-2).

[0473] Input: Text data uploaded by the user

[0474] Processing content: Tokenization, format conversion

[0475] Output: Preprocessed text data

[0476] Step 4:

[0477] The server uses the preprocessed visual, audio, and text data to train a generative model, which is trained using machine learning algorithms such as GAN and Transformer to generate visual and audio information to recreate the talent in their prime.

[0478] Input: Preprocessed visual data, audio data, and text data

[0479] Processing content: Training of generative models (GAN and Transformer)

[0480] Output: Generated visual and audio information

[0481] Step 5:

[0482] The server transmits the generated visual and audio information to the user's device, which then presents the information to the user. The user can then view the generated visual and audio information using a smartphone or head-mounted display and provide emotional feedback.

[0483] Input: Generated visual and audio information

[0484] Processing: Presenting information and collecting emotional feedback

[0485] Output: User's emotional feedback

[0486] Step 6:

[0487] The emotion engine analyzes user feedback, including facial expressions and tone of voice, and optimizes the generated visual and audio information based on the results. Feedback includes the user's facial expressions and tone of voice, and by analyzing these, it adjusts the output of the generative model.

[0488] Input: User's emotional feedback

[0489] Processing content: Analysis of emotional feedback and optimization of generated information

[0490] Output: Optimized visual and audio information

[0491] Step 7:

[0492] The server stores the optimized visual and audio information as non-fungible tokens (NFTs), a process that involves creating digital signatures and recording them on the blockchain.

[0493] Input: Optimized visual and audio information

[0494] Processing: Save as NFT, create digital signature, record on blockchain

[0495] Output: Visual and audio information stored as NFTs

[0496] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0497] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0498] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0499] [Second embodiment]

[0500] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0501] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0502] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0503] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0504] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0505] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0506] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0507] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0508] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0509] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0510] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0511] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0512] The system of the present invention collects visual, audio, and textual information, preprocesses the data, and then trains a generative model to generate visual and audio information that recreates the talent's appearance in their prime. The generated information is stored as an NFT, which can retain its value for a long period of time.

[0513] server

[0514] Data collection

[0515] The server receives visual, audio, and textual information uploaded by users, including talent images, video files, audio files, and social media posts.

[0516] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[0517] Data Preprocessing

[0518] The server analyzes the facial features of the received visual information, denoises the audio information, and tokenizes the text information.

[0519] Example: A server extracts facial features from a movie clip, removes noise from an audio file, or tokenizes and extracts features from a social media post.

[0520] Training a generative model

[0521] The server uses the preprocessed data to train a generative model (e.g., GAN or Transformer), which is used to integrate and reproduce the talent's visual, vocal, and thought patterns.

[0522] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[0523] AI Talent Generation

[0524] The server uses the trained generative model to generate visual and audio information for the talent based on the specified scenario.

[0525] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[0526] NFT conversion

[0527] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[0528] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[0529] Terminal

[0530] Entering and Submitting Data

[0531] The terminal checks for errors in the visual, audio, and text information entered by the user and sends the information to the server.

[0532] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[0533] Receiving and displaying the generated results

[0534] The terminal receives the generated visual and audio information from the server and displays it to the user.

[0535] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[0536] User

[0537] Talent selection and data provision

[0538] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[0539] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[0540] Check the generated results and request corrections

[0541] The user reviews the generated talent's visual and audio information and requests modifications from the server if necessary.

[0542] Example: A marketing person reviews the generated video and requests the server to correct specific facial expressions or lines.

[0543] final use

[0544] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[0545] Example: A marketer distributes the final video for an advertising campaign.

[0546] The system of the present invention makes it possible to execute a high-quality and consistent advertising strategy while perpetually maintaining the market value of a talent even after the talent has passed its prime.

[0547] The processing flow will be explained below.

[0548] Program processing flow

[0549] server

[0550] Step 1:

[0551] The server receives visual, audio, and textual information uploaded by users, including celebrity images, video files, audio files, and social media posts.

[0552] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[0553] Step 2:

[0554] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[0555] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[0556] Step 3:

[0557] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[0558] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[0559] Step 4:

[0560] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[0561] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[0562] Step 5:

[0563] The server uses the preprocessed data to train a generative model, integrating visual, audio, and text information to build a generative model that can recreate the talent in their prime.

[0564] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[0565] Step 6:

[0566] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[0567] Example: A server generates video and audio of an actor introducing a new product based on a specific scenario.

[0568] Step 7:

[0569] The server creates a digital signature and records it on the blockchain to store the generated visual and audio information as an NFT.

[0570] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[0571] Terminal

[0572] Step 1:

[0573] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[0574] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[0575] Step 2:

[0576] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[0577] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[0578] User

[0579] Step 1:

[0580] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[0581] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[0582] Step 2:

[0583] The user can check the generated visual and audio information of the talent and request corrections from the server as necessary, thereby creating optimal advertising materials.

[0584] Example: A marketing person reviews the generated video and requests the server to correct specific facial expressions or lines.

[0585] Step 3:

[0586] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[0587] Example: A marketer distributes the final video for an advertising campaign.

[0588] The above is a specific processing flow of the present invention, which makes it possible to execute a high-quality and consistent advertising strategy while perpetually maintaining the market value of a talent even after they have passed their prime.

[0589] Example 1

[0590] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0591] Conventional digital talent reproduction technologies rely on a single source of information (such as visual information only or audio information only), making it difficult to reproduce a talent's peak performance. Furthermore, there is no way to preserve the value of the generated digital data over the long term. This makes it difficult to implement high-quality, consistent advertising strategies.

[0592] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0593] In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative AI model using the preprocessed information, means for generating visual and audio information using the trained generative AI model, and means for digitally signing the generated visual and audio information and recording it on a blockchain as a non-fungible token. This makes it possible to reproduce a talent's peak years with high accuracy and preserve the value of the generated digital data for a long period of time.

[0594] "Visual information" refers to information that can be seen and recognized with the eyes, such as images and videos.

[0595] "Audio information" refers to information that can be heard by the ears, such as audio data or recording files.

[0596] "Text information" refers to information such as sentences and character data, the contents of which can be understood by reading it.

[0597] "Preprocessing" refers to a series of processes that convert collected information into a form suitable for input into a generative model, such as parsing, formatting, and tokenization.

[0598] "Generative AI model" refers to an artificial intelligence algorithm that is trained using collected and preprocessed data to generate new data (visual and audio information) based on specific inputs.

[0599] A "non-fungible token (NFT)" refers to a unique digital token that can prove ownership of a digital asset using blockchain technology.

[0600] "Digital Signature" refers to a cryptographic signature created to prove that digital data has not been tampered with.

[0601] "Blockchain" refers to a technology that maintains a series of records of digital transactions and manages them in a distributed manner across multiple computers.

[0602] The system of the present invention collects visual, audio, and text information, preprocesses the data, and then trains a generative AI model to generate visual and audio information that recreates the talent's appearance in their prime. The generated information is stored as a non-fungible token (NFT), which can retain its value for a long period of time.

[0603] server

[0604] The server uses HTTP requests to receive visual, audio, and textual information uploaded by users, such as images, video files, audio files, social media posts, etc. The server then validates the data and stores it in a database.

[0605] When preprocessing visual information, we use libraries such as OpenCV and Dlib to analyze facial features. When preprocessing audio information, we use audio processing libraries such as LibROSA to remove noise. When preprocessing text information, we use natural language processing libraries such as SpaCy and NLTK to tokenize and extract features.

[0606] TensorFlow and PyTorch are used to train generative AI models (e.g., GANs and Transformers) using the preprocessed data, which then builds highly accurate models that integrate the talent's appearance, voice, and thought patterns.

[0607] Based on the trained generative AI model, the server generates visual and audio information according to prompts given by the client. For example, a prompt such as "Generate a video in which actor X from the 1990s introduces new product Y. Product Y's features are A and B, and actor X should introduce the product in an enthusiastic and friendly tone."

[0608] The generated visual and audio information is digitally signed and recorded on the blockchain as a non-fungible token (NFT), using blockchain technologies such as Ethereum and Flow.

[0609] Terminal

[0610] The device checks the visual, audio, and text information entered by the user for errors and sends it to the server in the appropriate format, for example, checking the format of a movie clip, checking the sampling rate of an interview audio, and ensuring that a social media post is properly formatted.

[0611] It receives generated visual and audio information from the server, analyzes the generated results, and displays them on the screen or plays them back to the user. The user can review the generated content and request corrections if necessary.

[0612] User

[0613] Users log in to the system, select a period of time they want to focus on for a talent, and upload visual, audio, and text information related to that period. For example, a marketer could select an actor's data from the 1990s and upload film clips, audio interviews, and social media posts from that period.

[0614] Users can check the visual and audio information of the generated talent and request the server to modify specific facial expressions, lines, etc. Finally, they can select the content they are satisfied with and use it in corporate advertising campaigns and promotions, enabling them to execute high-quality, consistent advertising strategies while accurately recreating the talent in their prime.

[0615] This system allows users to easily create, store, and use digital information about celebrities, even after they have passed their prime, while maintaining their market value in perpetuity.

[0616] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0617] Step 1: Enter and submit data

[0618] Subject: Terminal

[0619] Specific operation: The user inputs visual, audio, and text information through the terminal. The terminal checks for errors in this data and converts it into the required format. Specifically, it checks the format of the movie clip and the audio sampling rate.

[0620] Input: User-uploaded film clips, interview audio files, social media posts

[0621] Output: Error checked and formatted data

[0622] Data processing: The device checks the format and data integrity and sends it to the server via an HTTP request.

[0623] Step 2: Receiving and storing data

[0624] Subject: Server

[0625] What happens: The server receives the HTTP request, checks the file format, verifies that it is in the correct format, and saves it to the database.

[0626] Input: Error-checked and formatted data sent from the terminal

[0627] Output: Visual, audio and text information stored in a database

[0628] Data processing: The server checks the data format and registers it in the database in a consistent state.

[0629] Step 3: Preprocessing the data

[0630] Subject: Server

[0631] Specific operation: The server uses libraries such as OpenCV and Dlib to extract facial features from visual information, audio processing libraries such as LibROSA to remove noise from audio data, and text information is tokenized and features are extracted using SpaCy and NLTK.

[0632] Input: Visual, audio, and text information stored in a database

[0633] Output: Preprocessed visual, audio, and text information

[0634] Data processing: The server analyzes visual information, reduces audio noise, tokenizes text, and extracts features.

[0635] Step 4: Training the generative AI model

[0636] Subject: Server

[0637] How it works: The server uses the preprocessed data to train a generative AI model (such as a GAN or Transformer). The training process is performed using TensorFlow or PyTorch to improve the model's accuracy.

[0638] Input: Preprocessed visual, audio, and text information

[0639] Output: A trained generative AI model

[0640] Data Computation: The server processes the data in batches and runs a training loop to improve the model's performance. It monitors the model's accuracy and loss function and adjusts hyperparameters as needed.

[0641] Step 5: Generate AI talent

[0642] Subject: Server

[0643] How it works: The server uses a trained generative AI model to generate visual and audio information based on prompts provided by the client.

[0644] Input: prompt, trained generative AI model

[0645] Output: Generated visual and audio information

[0646] Data calculation: The server analyzes the given prompt and inputs it into the generative AI model, which generates visual and audio information and retrieves that data.

[0647] Step 6: NFT

[0648] Subject: Server

[0649] How it works: The server digitally signs the generated visual and audio information and records it on the blockchain as a non-fungible token (NFT). It uses blockchain technologies such as Ethereum and Flow.

[0650] Input: Generated visual and audio information

[0651] Output: NFT recorded on the blockchain

[0652] Data processing: The server creates a digital signature and records ownership information on the blockchain.

[0653] Step 7: Receive and display the generated results

[0654] Subject: Terminal

[0655] Specific operation: The device receives the visual and audio information generated by the server and displays it to the user. The generated results are analyzed and displayed on the screen or played back as audio.

[0656] Input: Generated visual and audio information sent from the server

[0657] Output: Visual and audio information displayed to the user

[0658] Data processing: The device analyzes the generated data and displays and plays it appropriately for the user.

[0659] Step 8: Check the results and request corrections

[0660] Subject: User

[0661] Specific operation: The user checks the generated visual and audio information and requests the server to correct specific facial expressions, lines, etc.

[0662] Input: Generated visual and audio information

[0663] Output: Correction request

[0664] Data processing: The user checks the generated results, clarifies any corrections, and sends a request to the server.

[0665] Through these processing steps, users can create high-quality digital representations of their talent and use the generated information in advertising campaigns and promotions.

[0666] (Application example 1)

[0667] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0668] Conventional methods have had the problem that once a celebrity's prime passes, their market value declines, limiting their use in advertising and promotions. Furthermore, it is difficult to generate high-quality visual and audio information, and there is no way to maintain its value over the long term. Furthermore, when applied to virtual stores, there is a lack of means to enable real-time interactive customer service.

[0669] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0670] In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for recording the generated information on a blockchain and creating a digital signature for maintaining its value over the long term, and means for reproducing the generated visual and audio information in a virtual environment based on a user-specified scenario and displaying it to the user. This makes it possible to provide interactive customer service in a virtual store while perpetually maintaining the market value of a talent even after they have passed their prime.

[0671] "Visual information" refers to images, video clips, and other visual data.

[0672] "Audio information" refers to sound files, recordings, and other audio data.

[0673] "Text information" refers to sentences, notes, text files, and other text data.

[0674] "Preprocessing" refers to the process of analyzing collected data, removing noise, and extracting features.

[0675] "Generative model" refers to a trained algorithm for generating visual and audio information using artificial intelligence techniques.

[0676] "Digital signature" refers to a cryptographic technique used to ensure the authenticity and authenticity of generated information.

[0677] "Blockchain" refers to a system that links information in a chain and uses distributed ledger technology to prevent data tampering.

[0678] A "virtual environment" refers to a virtual space or situation generated by computer simulation.

[0679] This invention is a system for realizing interactive customer service in a virtual environment, and is composed of three elements: a server, a terminal, and a user.

[0680] server

[0681] The server collects visual, audio, and text information and preprocesses it. Specifically, the following processes are performed:

[0682] 1. Data Collection

[0683] The server receives visual information (e.g., images, video clips), audio information (e.g., audio recordings), and text information (e.g., social media posts) uploaded by users.

[0684] 2. Data Preprocessing

[0685] The received visual information is analyzed for facial features using an image processing library (e.g., OpenCV).

[0686] For audio information, an audio processing library is used to remove noise.

[0687] For text information, tokenization is performed using a natural language processing library (e.g., Transformers).

[0688] 3. Training the generative model

[0689] The pre-processed data is used to train a generative AI model (e.g., GAN or Transformer) that integrates the talent's visual, vocal, and thought patterns.

[0690] 4. AI Talent Generation

[0691] A trained generative model is used to generate visual and audio information based on a user-specified scenario.

[0692] 5. NFT conversion

[0693] The generated visual and audio information is recorded on the blockchain, a digital signature is created and stored as an NFT.

[0694] Terminal

[0695] The terminal has the following features:

[0696] 1. Enter and submit data

[0697] The visual, audio, and text information entered by the user is checked for errors and then sent to the server.

[0698] 2. Receiving and displaying the generated results

[0699] The system receives visual and audio information generated by the server and displays it to the user. Specifically, it plays video and audio of an AI talent introducing products in a virtual store environment.

[0700] User

[0701] The user performs the following operations:

[0702] 1. Talent selection and data provision

[0703] Users log in, select a talent's prime period, and upload visual, audio, and text information related to that period.

[0704] 2. Check the generated results and request corrections

[0705] Check the visual and audio information of the generated AI talent and request corrections from the server if necessary.

[0706] 3. End use

[0707] Finally, the content that satisfies the customer is selected and used for interactive customer service in the virtual store.

[0708] Specific examples

[0709] For example, a marketer could collect data on popular actors from the 1990s and generate a scenario to introduce a new product in a virtual store.

[0710] "Generate a scenario in which a popular actor from the 1990s appears in a virtual store and gives a detailed explanation of new products."

[0711] This system enables interactive customer service in a virtual store while perpetually maintaining the market value of talent even after they have passed their prime. The hardware used is a server and end-user devices, and the specific software includes OpenCV, Transformers, audio processing libraries, blockchain technology, etc.

[0712] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0713] Step 1:

[0714] The server collects visual, audio, and text information, including images, video clips, audio recordings, and social media posts uploaded by users. Specific data formats used include JPEG, MP4, MP3, and TXT. The received data is then temporarily stored in the server's database.

[0715] Step 2:

[0716] The server preprocesses the collected visual, audio, and text information. Specifically, for visual information, OpenCV is used to analyze facial features and a facial recognition algorithm is used to extract facial landmarks (eyes, nose, mouth, etc.). For audio information, an audio processing library is used to remove noise and convert it into clear audio data. For text information, the Transformers library is used to tokenize the data and convert each word or phrase into a vector representation. This preprocessed data is compiled into a training dataset and used in the next stage.

[0717] Step 3:

[0718] The server uses the preprocessed data to train a generative model. Specifically, the training dataset is input into a GAN (generative adversarial network) or Transformer model, and the model parameters are optimized. This results in a generative AI model capable of integrating and reproducing the talent's visual, vocal, and thought patterns. Model training is performed efficiently using a GPU to increase computational speed.

[0719] Step 4:

[0720] The server generates visual and audio information using a trained generative model. Based on a scenario specified by the user, a prompt sentence, such as "A celebrity will introduce a new product in a virtual store," is input into the generative model. Based on this prompt sentence, the model generates the celebrity's visual information (e.g., facial and body movements) and audio information (e.g., audio explaining the product). The generated data is temporarily stored in the server's storage.

[0721] Step 5:

[0722] The server stores the generated visual and audio information as an NFT. Specifically, it creates a digital signature for the generated data and records the information using blockchain technology. This guarantees the uniqueness of the generated data, making it possible to maintain its reliability and authenticity over the long term.

[0723] Step 6:

[0724] The device receives and displays the generated results. Specifically, it receives the generated visual and audio information sent from the server and displays and plays it for the user. For example, it plays video and audio of an AI talent introducing a product in a simulated virtual store environment. Based on this, the user can understand the details of the product and consider purchasing it.

[0725] Step 7:

[0726] The user checks the generated results and requests corrections. Specifically, the user requests the server to make minor corrections to the generated video and audio (e.g., changing specific facial expressions or lines). The server receives this request, makes corrections using the generative model again, and provides a satisfactory result.

[0727] Step 8:

[0728] Users can then select the content they are most satisfied with and use it in the virtual store. The generated AI talent's video and audio can then be used in actual promotions and advertising campaigns. This provides users with an interactive customer service experience and effectively communicates the appeal of the products.

[0729] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0730] The system of the present invention collects visual, audio, and text information, preprocesses the data, and then trains a generative model to generate visual and audio information that recreates the celebrity's appearance in their prime. Furthermore, by incorporating an emotion engine that recognizes user emotions and incorporating user feedback into the generation process, more optimal content generation is achieved. Furthermore, the generated information is stored as an NFT, which can maintain its value for a long period of time.

[0731] server

[0732] Data collection

[0733] The server receives visual, audio, and textual information uploaded by users, including talent images, video files, audio files, and social media posts.

[0734] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[0735] Data Preprocessing

[0736] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[0737] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[0738] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[0739] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[0740] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[0741] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[0742] Training a generative model

[0743] The server uses the preprocessed data to train a generative model (e.g., GAN or Transformer), which is used to integrate and reproduce the talent's visual, vocal, and thought patterns.

[0744] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[0745] AI Talent Generation

[0746] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[0747] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[0748] Utilizing the Emotion Engine

[0749] The server adjusts the output of the generative model using an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's feedback and optimizes the generated visual and audio information based on the results.

[0750] Example: The server recognizes the user's emotions in real time and incorporates that feedback into the generative model.

[0751] NFT conversion

[0752] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[0753] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[0754] Terminal

[0755] Entering and Submitting Data

[0756] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[0757] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[0758] Receiving and displaying the generated results

[0759] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[0760] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[0761] Feedback collection with emotion engine

[0762] The terminal transmits the emotions of the user while viewing the generated content to the emotion engine in real time.

[0763] Example: The device analyzes the user's facial expressions and speech while watching, and sends the emotional data to the server's emotion engine.

[0764] User

[0765] Talent selection and data provision

[0766] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[0767] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[0768] Check the generated results and request corrections

[0769] The user can review the visual and audio information of the generated talent, request corrections from the server if necessary, and provide feedback using the emotion engine to optimize the generation process.

[0770] Example: A marketer reviews the generated video and requests the server to modify specific facial expressions or lines, and the emotion engine learns from user feedback.

[0771] final use

[0772] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[0773] Example: A marketer distributes the final video for an advertising campaign.

[0774] The system of the present invention makes it possible to execute a high-quality, consistent advertising strategy that is in line with user emotions while perpetually maintaining the market value of a talent even after the talent has passed its prime.

[0775] The processing flow will be explained below.

[0776] Program processing flow

[0777] server

[0778] Step 1:

[0779] The server receives visual, audio, and textual information uploaded by users, including celebrity images, video files, audio files, and social media posts.

[0780] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[0781] Step 2:

[0782] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[0783] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[0784] Step 3:

[0785] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[0786] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[0787] Step 4:

[0788] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[0789] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[0790] Step 5:

[0791] The server uses the preprocessed data to train a generative model, integrating visual, audio, and text information to build a generative model that can recreate the talent in their prime.

[0792] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[0793] Step 6:

[0794] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[0795] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[0796] Step 7:

[0797] The server uses an emotion engine to analyze the user's emotions and reflects them as feedback to the generative model, thereby improving the quality of the generated content.

[0798] Example: The server collects data on the user's emotions and reflects it in the generation process.

[0799] Step 8:

[0800] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[0801] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[0802] Terminal

[0803] Step 1:

[0804] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[0805] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[0806] Step 2:

[0807] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[0808] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[0809] Step 3:

[0810] The device transmits emotional data in real time to a server while the user is viewing the generated content, and emotion recognition software is used to analyze emotions such as facial expressions and speech.

[0811] Example: The device analyzes the user's facial expressions and speech while watching and sends the emotional data to the server.

[0812] User

[0813] Step 1:

[0814] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[0815] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[0816] Step 2:

[0817] The user can review the visual and audio information of the generated talent, request corrections from the server if necessary, and provide feedback using the emotion engine to optimize the generation process.

[0818] Example: A marketer reviews the generated video and requests the server to modify specific facial expressions or lines, and the emotion engine learns from user feedback.

[0819] Step 3:

[0820] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[0821] Example: A marketer distributes the final video for an advertising campaign.

[0822] The above is a specific processing flow of the present invention. This system makes it possible to execute a high-quality, consistent advertising strategy that is in tune with the user's emotions while perpetually maintaining the market value of a talent even after they have passed their prime.

[0823] Example 2

[0824] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0825] In today's digital media environment, there is a need to generate high-quality, consistent visual and audio content while reusing the assets of past talent. However, current technologies lack a method for consistently processing visual, audio, and text information and training effective generative models, resulting in inconsistent quality of generated content. Furthermore, the lack of a function for adjusting content based on user emotions makes it difficult to provide an optimal entertainment experience.

[0826] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for extracting facial features from the visual information of the preprocessed information, means for denoising the preprocessed audio information, means for tokenizing and analyzing the preprocessed text information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for adjusting the generated visual and audio information based on user emotional feedback, and means for storing the generated information as an NFT. This enables effective reuse of past talent assets and the generation of high-quality, consistent visual and audio content adjusted based on user emotions.

[0827] "Visual information" is information that is presented in a form that can be seen with the eyes, such as an image or video file.

[0828] "Audio information" refers to information that is expressed in a form that can be heard through the ears, such as audio files or recorded data.

[0829] "Text information" refers to information that is written in a readable form, such as written characters or sentences.

[0830] "Preprocessing" refers to the process of preparing collected raw data in a form that is easier to analyze, and includes processes such as noise removal and tokenization.

[0831] "Facial features" refers to information extracted from visual information about a person's facial shape and characteristic points such as the eyes, nose, and mouth.

[0832] "Noise reduction" is a process for removing unnecessary background sounds and interference sounds contained in audio information.

[0833] "Tokenization" is a process of dividing text information into small units such as words and phrases and analyzing their meanings and relationships.

[0834] A "generative model" is an algorithm used to generate new visual or audio information based on input data, and includes GANs and Transformers.

[0835] A "trained generative model" is a generative algorithm that has been optimized using preprocessed data and is used to generate new content based on a specified scenario.

[0836] "Emotional feedback" is a process of collecting evaluations and opinions based on the user's emotional state and adjusting the generated content based on that information.

[0837] An "NFT" is a token that uses blockchain technology to prove ownership of digital data and represents a one-of-a-kind digital asset.

[0838] This system generates high-quality visual and audio information by integrating visual, audio, and text information, preprocessing the data, and training a generative model. The generated information is stored as NFTs, which retain their value over the long term. This system provides optimized content based on specific scenarios and user emotional feedback.

[0839] server

[0840] Data collection

[0841] The server receives visual, audio, and text information uploaded by the user through the device. Specifically, it receives HTTP requests and saves image files and text data in storage.

[0842] Example: A server receives and stores actor film clips, interview audio files, and social media posts sent by a marketer.

[0843] Data Preprocessing

[0844] The server preprocesses the collected visual information and extracts facial features using a facial recognition algorithm such as FaceNet. It also preprocesses the audio information by removing noise using FFmpeg or Audacity. It then tokenizes and analyzes the text information using natural language processing libraries such as NLTK and spaCy.

[0845] Examples: Extracting actors' facial features from movie clips, removing background noise from interview audio, and tokenizing and analyzing social media posts.

[0846] Training a generative model

[0847] The server uses the preprocessed data to train a generative model such as a GAN or Transformer, using TensorFlow or PyTorch to integrate facial features, audio data, and text data to optimize the model.

[0848] Example: Training a generative model of AI talent using facial features, voice features, and thought patterns derived from text data.

[0849] AI Talent Generation

[0850] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a given scenario, inputs a prompt (e.g., "Introducing a new product") into the model, and stores the generated video and audio.

[0851] Example: Based on a specific product promotion scenario, generate video and audio of an actor introducing a product.

[0852] Utilizing the Emotion Engine

[0853] The server uses an emotion engine to analyze user feedback and adjust the generated content. The emotion engine recognizes user emotions in real time and fine-tunes the generative model based on them.

[0854] Example: Collecting the user's emotions as feedback and adjusting the actor's facial expressions based on that data.

[0855] NFT conversion

[0856] The server stores the generated talent's visual and audio information as an NFT, which is recorded on the Ethereum blockchain with a digital signature, making it a unique digital asset.

[0857] Example: The generated video and audio of an actor is registered as an NFT, and its ownership information is recorded on the blockchain.

[0858] Terminal

[0859] Entering and Submitting Data

[0860] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits it to the server.

[0861] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[0862] Receiving and displaying the generated results

[0863] The terminal receives the generated content from the server and displays it to the user, allowing the user to review the results and make correction requests if necessary.

[0864] Example: Display a video of an actor introducing a new product sent from the server and play an audio file.

[0865] Feedback collection with emotion engine

[0866] While the user is viewing the generated content, the device analyzes facial expressions and speech in real time and sends the results to the emotion engine.

[0867] Example: Analyze the facial expressions and speech of the user while watching, and send the emotional data to the emotion engine on the server.

[0868] User

[0869] Talent selection and data provision

[0870] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[0871] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[0872] Check the generated results and request corrections

[0873] The user can review the generated visual and audio information, request corrections from the server, and even provide feedback using the emotion engine to optimize the generation process.

[0874] Example: The generated video is reviewed, and specific facial expressions or lines are corrected by the server, and the emotion engine learns from the user's feedback.

[0875] final use

[0876] The user ultimately selects the content they are satisfied with and uses it for advertising campaigns and promotions.

[0877] Example: A marketer distributes the final video for an advertising campaign.

[0878] Prompt Sentence Examples

[0879] "Generate a new product promotional video using the face and voice of a famous actor from the 1980s. Use uploaded film clips and interview audio."

[0880] "Please use the user's emotional data as a reference to optimize the facial expressions of the actors in the generated video to make them appear happier."

[0881] The above is an embodiment of the present invention.

[0882] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0883] Step 1: Data collection

[0884] The server receives visual, audio, and text information uploaded by users through their devices. Specifically, it receives HTTP requests and saves image files and text data in storage. The input to this process is the data provided by the user, and the output is the saved raw data. Specifically, the server receives and saves actor film clips, interview audio files, and social media posts sent by marketers.

[0885] Step 2: Data preprocessing (visual information)

[0886] The server preprocesses the received visual information and extracts facial features using a facial recognition algorithm such as FaceNet. The input to this process is the collected visual information, and the output is the extracted facial feature information. Specifically, the server uses OpenCV to analyze frames from a movie clip, detect facial regions, and extract feature points (eyes, nose, mouth, etc.).

[0887] Step 3: Data preprocessing (audio information)

[0888] The server uses FFmpeg and Audacity to preprocess the audio information and remove noise. The input of this process is the collected audio information, and the output is clear audio data with noise removed. Specifically, the server uses FFmpeg to read the audio file, apply a filter to remove noise, and generate clear audio data.

[0889] Step 4: Data preprocessing (text information)

[0890] The server preprocesses the text information and performs tokenization using a natural language processing library (NLTK or spaCy). The input to this process is the collected text information, and the output is tokenized text data. Specifically, the server reads the text data, splits it into tokens, extracts keywords and important phrases, and stores them in a database.

[0891] Step 5: Training the generative model

[0892] The server uses the preprocessed data to train a generative model such as a GAN or Transformer. The input for this process is the preprocessed visual, audio, and text information, and the output is a trained generative model. Specifically, the server uses TensorFlow or PyTorch to train the model using facial features, audio data, and text data as input, and saves the trained model.

[0893] Step 6: Generate AI talent

[0894] The server uses a trained generative model to generate visual and audio information for an AI talent based on a specified scenario. The input to this process is the scenario information and the trained generative model, and the output is the generated visual and audio information. Specifically, the server inputs a prompt sentence (e.g., "Introducing a new product") into the model and saves the generated video and audio.

[0895] Step 7: Leverage your emotional engine

[0896] The device analyzes the user's facial expressions and speech in real time while watching the generated content and sends the results to the emotion engine. The input for this process is the user's facial expression data and voice data, and the output is emotional feedback. Specifically, the device's camera and microphone are used to collect the user's facial expression data and voice data, and the data is sent to the emotion engine in real time.

[0897] The server uses an emotion engine to analyze user feedback and adjusts the generated content by incorporating the results into the generative model. The input to this process is the emotional feedback sent from the device, and the output is the adjusted generated content. Specifically, the emotion engine identifies the user's emotions and optimizes the generated video and audio based on them.

[0898] Step 8: NFT

[0899] The server stores the generated visual and audio information of the talent as an NFT. The input of this process is the generated visual and audio information, and the output is an NFT recorded on the blockchain. Specifically, the server digitally signs the generated data and records it on a blockchain network such as Ethereum.

[0900] (Application example 2)

[0901] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0902] In the prior art, there was no effective method for generating visual and audio information to recreate a specific celebrity's peak performance, and it was difficult to reflect user emotional feedback in real time. Furthermore, there was insufficient protection of the value of the generated content and proof of ownership, making it difficult to continue providing high-quality content that satisfies users. The present invention aims to solve these problems.

[0903] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for storing the generated information as a non-fungible token, and means for collecting and analyzing user feedback using an emotion engine to allow users to customize the generated visual and audio information and reflecting the feedback in the generation process. This makes it possible to continue providing high-quality, customizable content and also makes it easier to preserve the value of the generated content and prove ownership.

[0904] "Visual information" is a general term for image and video data collected from users and other sources.

[0905] "Audio Information" refers collectively to audio data collected from users and other sources.

[0906] "Text Information" refers collectively to written or typed character information collected from users or other sources.

[0907] "Preprocessing" is the process of formatting collected visual, audio, and text information to make it easier to analyze.

[0908] A "generative model" is a machine learning or deep learning algorithm that uses collected and preprocessed data to generate new visual or audio information.

[0909] A "non-fungible token" is a digital asset with unique value based on a blockchain, typically used for proof of ownership and store of value purposes.

[0910] An "emotion engine" is an algorithm and system that analyzes users' emotions in real time and collects and reflects the results as feedback.

[0911] "User feedback" refers to the user's impressions and reactions to the generated visual and audio information, and requests for modifications based on the impressions and reactions.

[0912] This invention provides a system that collects visual, audio, and text information, preprocesses them, and then generates visual and audio information using a generative model. The system includes a means for storing the generated information as a non-fungible token (NFT). The system also uses an emotion engine to allow users to customize the generated visual and audio information, collects and analyzes user feedback, and incorporates that feedback into the generation process.

[0913] System configuration and operation overview

[0914] 1. Visual information collection and preprocessing:

[0915] The server receives image and video files uploaded by users, such as old movie clips or interview footage.

[0916] Use OpenCV to resize and reshape the image and extract facial features.

[0917] 2. Acquisition and preprocessing of audio information:

[0918] The server receives audio files uploaded by users, for example, interview audio data.

[0919] Use pydub to convert the audio data to mono and format it in a way that is suitable for the speech recognition model.

[0920] 3. Text information collection and preprocessing:

[0921] The server receives text information uploaded by the user, such as a social media post or a script.

[0922] Text information is tokenized and converted into a format that can be input into natural language processing models such as GPT-2.

[0923] 4. Training the generative model and generating visual and audio information:

[0924] The server trains a generative model (e.g., GAN, Transformer) using the preprocessed data.

[0925] A trained generative model is used to generate visual and audio information.

[0926] 5. Feedback collection and analysis using emotion engine:

[0927] The terminal presents the generated visual and audio information to the user and collects the user's emotional feedback in real time.

[0928] The emotion engine analyzes the user's facial expressions and voice and sends the feedback to the server.

[0929] 6. Optimization of generated information and NFTization:

[0930] The server optimizes the generation process based on user feedback.

[0931] The final content generated is stored as a non-fungible token (NFT) to preserve value and prove ownership.

[0932] Specific examples

[0933] If an advertising agency wants to use a famous celebrity from the 1990s to promote a new product, they can use this system. They can upload old movie clips, interview audio, social media posts, etc. to train a generative model. The system then generates an advertising video, and users can provide real-time feedback via smartphone or head-mounted display. The final optimized advertising video is stored as an NFT, preserving ownership and value.

[0934] Example prompt sentence:

[0935] "Generate an advertising video for new product X featuring famous 90s actor A. Use the following information: movie clips, interview audio, and social media posts."

[0936] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0937] Step 1:

[0938] The server collects visual information from users. Specifically, users upload image or video files, such as old movie clips or interview footage. The server receives these files, resizes and reshapes the images using OpenCV, and extracts facial features.

[0939] Input: User-uploaded film clips and interview footage

[0940] Processing: Resize, shape, and extract facial features using OpenCV

[0941] Output: Preprocessed visual data

[0942] Step 2:

[0943] The server collects audio information from users, specifically, by uploading interview recordings or other audio files. The server receives these audio files and uses pydub to convert them to mono and remove background noise.

[0944] Input: Audio file uploaded by the user

[0945] Processing: Mono conversion and noise removal using pydub

[0946] Output: Preprocessed audio data

[0947] Step 3:

[0948] The server collects text information from users. Specifically, users upload text data such as social media posts and scripts. The server receives this text information, tokenizes it, and converts it into a format suitable for input into a natural language processing model (such as GPT-2).

[0949] Input: Text data uploaded by the user

[0950] Processing content: Tokenization, format conversion

[0951] Output: Preprocessed text data

[0952] Step 4:

[0953] The server uses the preprocessed visual, audio, and text data to train a generative model, which is trained using machine learning algorithms such as GAN and Transformer to generate visual and audio information to recreate the talent in their prime.

[0954] Input: Preprocessed visual data, audio data, and text data

[0955] Processing content: Training of generative models (GAN and Transformer)

[0956] Output: Generated visual and audio information

[0957] Step 5:

[0958] The server transmits the generated visual and audio information to the user's device, which then presents the information to the user. The user can then view the generated visual and audio information using a smartphone or head-mounted display and provide emotional feedback.

[0959] Input: Generated visual and audio information

[0960] Processing: Presenting information and collecting emotional feedback

[0961] Output: User's emotional feedback

[0962] Step 6:

[0963] The emotion engine analyzes user feedback, including facial expressions and tone of voice, and optimizes the generated visual and audio information based on the results. Feedback includes the user's facial expressions and tone of voice, and by analyzing these, it adjusts the output of the generative model.

[0964] Input: User's emotional feedback

[0965] Processing content: Analysis of emotional feedback and optimization of generated information

[0966] Output: Optimized visual and audio information

[0967] Step 7:

[0968] The server stores the optimized visual and audio information as non-fungible tokens (NFTs), a process that involves creating digital signatures and recording them on the blockchain.

[0969] Input: Optimized visual and audio information

[0970] Processing: Save as NFT, create digital signature, record on blockchain

[0971] Output: Visual and audio information stored as NFTs

[0972] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0973] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0974] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0975] [Third embodiment]

[0976] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0977] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0978] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0979] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0980] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0981] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0982] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0983] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0984] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0985] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0986] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0987] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0988] The system of the present invention collects visual, audio, and textual information, preprocesses the data, and then trains a generative model to generate visual and audio information that recreates the talent's appearance in their prime. The generated information is stored as an NFT, which can retain its value for a long period of time.

[0989] server

[0990] Data collection

[0991] The server receives visual, audio, and textual information uploaded by users, including talent images, video files, audio files, and social media posts.

[0992] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[0993] Data Preprocessing

[0994] The server analyzes the facial features of the received visual information, denoises the audio information, and tokenizes the text information.

[0995] Example: A server extracts facial features from a movie clip, removes noise from an audio file, or tokenizes and extracts features from a social media post.

[0996] Training a generative model

[0997] The server uses the preprocessed data to train a generative model (e.g., GAN or Transformer), which is used to integrate and reproduce the talent's visual, vocal, and thought patterns.

[0998] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[0999] AI Talent Generation

[1000] The server uses the trained generative model to generate visual and audio information for the talent based on the specified scenario.

[1001] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[1002] NFT conversion

[1003] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[1004] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[1005] Terminal

[1006] Entering and Submitting Data

[1007] The terminal checks for errors in the visual, audio, and text information entered by the user and sends the information to the server.

[1008] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[1009] Receiving and displaying the generated results

[1010] The terminal receives the generated visual and audio information from the server and displays it to the user.

[1011] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[1012] User

[1013] Talent selection and data provision

[1014] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[1015] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[1016] Check the generated results and request corrections

[1017] The user reviews the generated talent's visual and audio information and requests modifications from the server if necessary.

[1018] Example: A marketing person reviews the generated video and requests the server to correct specific facial expressions or lines.

[1019] final use

[1020] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[1021] Example: A marketer distributes the final video for an advertising campaign.

[1022] The system of the present invention makes it possible to execute a high-quality and consistent advertising strategy while perpetually maintaining the market value of a talent even after the talent has passed its prime.

[1023] The processing flow will be explained below.

[1024] Program processing flow

[1025] server

[1026] Step 1:

[1027] The server receives visual, audio, and textual information uploaded by users, including celebrity images, video files, audio files, and social media posts.

[1028] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[1029] Step 2:

[1030] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[1031] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[1032] Step 3:

[1033] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[1034] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[1035] Step 4:

[1036] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[1037] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[1038] Step 5:

[1039] The server uses the preprocessed data to train a generative model, integrating visual, audio, and text information to build a generative model that can recreate the talent in their prime.

[1040] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[1041] Step 6:

[1042] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[1043] Example: A server generates video and audio of an actor introducing a new product based on a specific scenario.

[1044] Step 7:

[1045] The server creates a digital signature and records it on the blockchain to store the generated visual and audio information as an NFT.

[1046] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[1047] Terminal

[1048] Step 1:

[1049] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[1050] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[1051] Step 2:

[1052] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[1053] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[1054] User

[1055] Step 1:

[1056] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[1057] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[1058] Step 2:

[1059] The user can check the generated visual and audio information of the talent and request corrections from the server as necessary, thereby creating optimal advertising materials.

[1060] Example: A marketing person reviews the generated video and requests the server to correct specific facial expressions or lines.

[1061] Step 3:

[1062] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[1063] Example: A marketer distributes the final video for an advertising campaign.

[1064] The above is a specific processing flow of the present invention, which makes it possible to execute a high-quality and consistent advertising strategy while perpetually maintaining the market value of a talent even after they have passed their prime.

[1065] Example 1

[1066] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1067] Conventional digital talent reproduction technologies rely on a single source of information (such as visual information only or audio information only), making it difficult to reproduce a talent's peak performance. Furthermore, there is no way to preserve the value of the generated digital data over the long term. This makes it difficult to implement high-quality, consistent advertising strategies.

[1068] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1069] In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative AI model using the preprocessed information, means for generating visual and audio information using the trained generative AI model, and means for digitally signing the generated visual and audio information and recording it on a blockchain as a non-fungible token. This makes it possible to reproduce a talent's peak years with high accuracy and preserve the value of the generated digital data for a long period of time.

[1070] "Visual information" refers to information that can be seen and recognized with the eyes, such as images and videos.

[1071] "Audio information" refers to information that can be heard by the ears, such as audio data or recording files.

[1072] "Text information" refers to information such as sentences and character data, the contents of which can be understood by reading it.

[1073] "Preprocessing" refers to a series of processes that convert collected information into a form suitable for input into a generative model, such as parsing, formatting, and tokenization.

[1074] "Generative AI model" refers to an artificial intelligence algorithm that is trained using collected and preprocessed data to generate new data (visual and audio information) based on specific inputs.

[1075] A "non-fungible token (NFT)" refers to a unique digital token that can prove ownership of a digital asset using blockchain technology.

[1076] "Digital Signature" refers to a cryptographic signature created to prove that digital data has not been tampered with.

[1077] "Blockchain" refers to a technology that maintains a series of records of digital transactions and manages them in a distributed manner across multiple computers.

[1078] The system of the present invention collects visual, audio, and text information, preprocesses the data, and then trains a generative AI model to generate visual and audio information that recreates the talent's appearance in their prime. The generated information is stored as a non-fungible token (NFT), which can retain its value for a long period of time.

[1079] server

[1080] The server uses HTTP requests to receive visual, audio, and textual information uploaded by users, such as images, video files, audio files, social media posts, etc. The server then validates the data and stores it in a database.

[1081] When preprocessing visual information, we use libraries such as OpenCV and Dlib to analyze facial features. When preprocessing audio information, we use audio processing libraries such as LibROSA to remove noise. When preprocessing text information, we use natural language processing libraries such as SpaCy and NLTK to tokenize and extract features.

[1082] TensorFlow and PyTorch are used to train generative AI models (e.g., GANs and Transformers) using the preprocessed data, which then builds highly accurate models that integrate the talent's appearance, voice, and thought patterns.

[1083] Based on the trained generative AI model, the server generates visual and audio information according to prompts given by the client. For example, a prompt such as "Generate a video in which actor X from the 1990s introduces new product Y. Product Y's features are A and B, and actor X should introduce the product in an enthusiastic and friendly tone."

[1084] The generated visual and audio information is digitally signed and recorded on the blockchain as a non-fungible token (NFT), using blockchain technologies such as Ethereum and Flow.

[1085] Terminal

[1086] The device checks the visual, audio, and text information entered by the user for errors and sends it to the server in the appropriate format, for example, checking the format of a movie clip, checking the sampling rate of an interview audio, and ensuring that a social media post is properly formatted.

[1087] It receives generated visual and audio information from the server, analyzes the generated results, and displays them on the screen or plays them back to the user. The user can review the generated content and request corrections if necessary.

[1088] User

[1089] Users log in to the system, select a period of time they want to focus on for a talent, and upload visual, audio, and text information related to that period. For example, a marketer could select an actor's data from the 1990s and upload film clips, audio interviews, and social media posts from that period.

[1090] Users can check the visual and audio information of the generated talent and request the server to modify specific facial expressions, lines, etc. Finally, they can select the content they are satisfied with and use it in corporate advertising campaigns and promotions, enabling them to execute high-quality, consistent advertising strategies while accurately recreating the talent in their prime.

[1091] This system allows users to easily create, store, and use digital information about celebrities, even after they have passed their prime, while maintaining their market value in perpetuity.

[1092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1093] Step 1: Enter and submit data

[1094] Subject: Terminal

[1095] Specific operation: The user inputs visual, audio, and text information through the terminal. The terminal checks for errors in this data and converts it into the required format. Specifically, it checks the format of the movie clip and the audio sampling rate.

[1096] Input: User-uploaded film clips, interview audio files, social media posts

[1097] Output: Error checked and formatted data

[1098] Data processing: The device checks the format and data integrity and sends it to the server via an HTTP request.

[1099] Step 2: Receiving and storing data

[1100] Subject: Server

[1101] What happens: The server receives the HTTP request, checks the file format, verifies that it is in the correct format, and saves it to the database.

[1102] Input: Error-checked and formatted data sent from the terminal

[1103] Output: Visual, audio and text information stored in a database

[1104] Data processing: The server checks the data format and registers it in the database in a consistent state.

[1105] Step 3: Preprocessing the data

[1106] Subject: Server

[1107] Specific operation: The server uses libraries such as OpenCV and Dlib to extract facial features from visual information, audio processing libraries such as LibROSA to remove noise from audio data, and text information is tokenized and features are extracted using SpaCy and NLTK.

[1108] Input: Visual, audio, and text information stored in a database

[1109] Output: Preprocessed visual, audio, and text information

[1110] Data processing: The server analyzes visual information, reduces audio noise, tokenizes text, and extracts features.

[1111] Step 4: Training the generative AI model

[1112] Subject: Server

[1113] How it works: The server uses the preprocessed data to train a generative AI model (such as a GAN or Transformer). The training process is performed using TensorFlow or PyTorch to improve the model's accuracy.

[1114] Input: Preprocessed visual, audio, and text information

[1115] Output: A trained generative AI model

[1116] Data Computation: The server processes the data in batches and runs a training loop to improve the model's performance. It monitors the model's accuracy and loss function and adjusts hyperparameters as needed.

[1117] Step 5: Generate AI talent

[1118] Subject: Server

[1119] How it works: The server uses a trained generative AI model to generate visual and audio information based on prompts provided by the client.

[1120] Input: prompt, trained generative AI model

[1121] Output: Generated visual and audio information

[1122] Data calculation: The server analyzes the given prompt and inputs it into the generative AI model, which generates visual and audio information and retrieves that data.

[1123] Step 6: NFT

[1124] Subject: Server

[1125] How it works: The server digitally signs the generated visual and audio information and records it on the blockchain as a non-fungible token (NFT). It uses blockchain technologies such as Ethereum and Flow.

[1126] Input: Generated visual and audio information

[1127] Output: NFT recorded on the blockchain

[1128] Data processing: The server creates a digital signature and records ownership information on the blockchain.

[1129] Step 7: Receive and display the generated results

[1130] Subject: Terminal

[1131] Specific operation: The device receives the visual and audio information generated by the server and displays it to the user. The generated results are analyzed and displayed on the screen or played back as audio.

[1132] Input: Generated visual and audio information sent from the server

[1133] Output: Visual and audio information displayed to the user

[1134] Data processing: The device analyzes the generated data and displays and plays it appropriately for the user.

[1135] Step 8: Check the results and request corrections

[1136] Subject: User

[1137] Specific operation: The user checks the generated visual and audio information and requests the server to correct specific facial expressions, lines, etc.

[1138] Input: Generated visual and audio information

[1139] Output: Correction request

[1140] Data processing: The user checks the generated results, clarifies any corrections, and sends a request to the server.

[1141] Through these processing steps, users can create high-quality digital representations of their talent and use the generated information in advertising campaigns and promotions.

[1142] (Application example 1)

[1143] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1144] Conventional methods have had the problem that once a celebrity's prime passes, their market value declines, limiting their use in advertising and promotions. Furthermore, it is difficult to generate high-quality visual and audio information, and there is no way to maintain its value over the long term. Furthermore, when applied to virtual stores, there is a lack of means to enable real-time interactive customer service.

[1145] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1146] In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for recording the generated information on a blockchain and creating a digital signature for maintaining its value over the long term, and means for reproducing the generated visual and audio information in a virtual environment based on a user-specified scenario and displaying it to the user. This makes it possible to provide interactive customer service in a virtual store while perpetually maintaining the market value of a talent even after they have passed their prime.

[1147] "Visual information" refers to images, video clips, and other visual data.

[1148] "Audio information" refers to sound files, recordings, and other audio data.

[1149] "Text information" refers to sentences, notes, text files, and other text data.

[1150] "Preprocessing" refers to the process of analyzing collected data, removing noise, and extracting features.

[1151] "Generative model" refers to a trained algorithm for generating visual and audio information using artificial intelligence techniques.

[1152] "Digital signature" refers to a cryptographic technique used to ensure the authenticity and authenticity of generated information.

[1153] "Blockchain" refers to a system that links information in a chain and uses distributed ledger technology to prevent data tampering.

[1154] A "virtual environment" refers to a virtual space or situation generated by computer simulation.

[1155] This invention is a system for realizing interactive customer service in a virtual environment, and is composed of three elements: a server, a terminal, and a user.

[1156] server

[1157] The server collects visual, audio, and text information and preprocesses it. Specifically, the following processes are performed:

[1158] 1. Data Collection

[1159] The server receives visual information (e.g., images, video clips), audio information (e.g., audio recordings), and text information (e.g., social media posts) uploaded by users.

[1160] 2. Data Preprocessing

[1161] The received visual information is analyzed for facial features using an image processing library (e.g., OpenCV).

[1162] For audio information, an audio processing library is used to remove noise.

[1163] For text information, tokenization is performed using a natural language processing library (e.g., Transformers).

[1164] 3. Training the generative model

[1165] The pre-processed data is used to train a generative AI model (e.g., GAN or Transformer) that integrates the talent's visual, vocal, and thought patterns.

[1166] 4. AI Talent Generation

[1167] A trained generative model is used to generate visual and audio information based on a user-specified scenario.

[1168] 5. NFT conversion

[1169] The generated visual and audio information is recorded on the blockchain, a digital signature is created and stored as an NFT.

[1170] Terminal

[1171] The terminal has the following features:

[1172] 1. Enter and submit data

[1173] The visual, audio, and text information entered by the user is checked for errors and then sent to the server.

[1174] 2. Receiving and displaying the generated results

[1175] The system receives visual and audio information generated by the server and displays it to the user. Specifically, it plays video and audio of an AI talent introducing products in a virtual store environment.

[1176] User

[1177] The user performs the following operations:

[1178] 1. Talent selection and data provision

[1179] Users log in, select a talent's prime period, and upload visual, audio, and text information related to that period.

[1180] 2. Check the generated results and request corrections

[1181] Check the visual and audio information of the generated AI talent and request corrections from the server if necessary.

[1182] 3. End use

[1183] Finally, the content that satisfies the customer is selected and used for interactive customer service in the virtual store.

[1184] Specific examples

[1185] For example, a marketer could collect data on popular actors from the 1990s and generate a scenario to introduce a new product in a virtual store.

[1186] "Generate a scenario in which a popular actor from the 1990s appears in a virtual store and gives a detailed explanation of new products."

[1187] This system enables interactive customer service in a virtual store while perpetually maintaining the market value of talent even after they have passed their prime. The hardware used is a server and end-user devices, and the specific software includes OpenCV, Transformers, audio processing libraries, blockchain technology, etc.

[1188] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1189] Step 1:

[1190] The server collects visual, audio, and text information, including images, video clips, audio recordings, and social media posts uploaded by users. Specific data formats used include JPEG, MP4, MP3, and TXT. The received data is then temporarily stored in the server's database.

[1191] Step 2:

[1192] The server preprocesses the collected visual, audio, and text information. Specifically, for visual information, OpenCV is used to analyze facial features and a facial recognition algorithm is used to extract facial landmarks (eyes, nose, mouth, etc.). For audio information, an audio processing library is used to remove noise and convert it into clear audio data. For text information, the Transformers library is used to tokenize the data and convert each word or phrase into a vector representation. This preprocessed data is compiled into a training dataset and used in the next stage.

[1193] Step 3:

[1194] The server uses the preprocessed data to train a generative model. Specifically, the training dataset is input into a GAN (generative adversarial network) or Transformer model, and the model parameters are optimized. This results in a generative AI model capable of integrating and reproducing the talent's visual, vocal, and thought patterns. Model training is performed efficiently using a GPU to increase computational speed.

[1195] Step 4:

[1196] The server generates visual and audio information using a trained generative model. Based on a scenario specified by the user, a prompt sentence, such as "A celebrity will introduce a new product in a virtual store," is input into the generative model. Based on this prompt sentence, the model generates the celebrity's visual information (e.g., facial and body movements) and audio information (e.g., audio explaining the product). The generated data is temporarily stored in the server's storage.

[1197] Step 5:

[1198] The server stores the generated visual and audio information as an NFT. Specifically, it creates a digital signature for the generated data and records the information using blockchain technology. This guarantees the uniqueness of the generated data, making it possible to maintain its reliability and authenticity over the long term.

[1199] Step 6:

[1200] The device receives and displays the generated results. Specifically, it receives the generated visual and audio information sent from the server and displays and plays it for the user. For example, it plays video and audio of an AI talent introducing a product in a simulated virtual store environment. Based on this, the user can understand the details of the product and consider purchasing it.

[1201] Step 7:

[1202] The user checks the generated results and requests corrections. Specifically, the user requests the server to make minor corrections to the generated video and audio (e.g., changing specific facial expressions or lines). The server receives this request, makes corrections using the generative model again, and provides a satisfactory result.

[1203] Step 8:

[1204] Users can then select the content they are most satisfied with and use it in the virtual store. The generated AI talent's video and audio can then be used in actual promotions and advertising campaigns. This provides users with an interactive customer service experience and effectively communicates the appeal of the products.

[1205] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1206] The system of the present invention collects visual, audio, and text information, preprocesses the data, and then trains a generative model to generate visual and audio information that recreates the celebrity's appearance in their prime. Furthermore, by incorporating an emotion engine that recognizes user emotions and incorporating user feedback into the generation process, more optimal content generation is achieved. Furthermore, the generated information is stored as an NFT, which can maintain its value for a long period of time.

[1207] server

[1208] Data collection

[1209] The server receives visual, audio, and textual information uploaded by users, including talent images, video files, audio files, and social media posts.

[1210] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[1211] Data Preprocessing

[1212] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[1213] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[1214] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[1215] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[1216] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[1217] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[1218] Training a generative model

[1219] The server uses the preprocessed data to train a generative model (e.g., GAN or Transformer), which is used to integrate and reproduce the talent's visual, vocal, and thought patterns.

[1220] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[1221] AI Talent Generation

[1222] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[1223] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[1224] Utilizing the Emotion Engine

[1225] The server adjusts the output of the generative model using an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's feedback and optimizes the generated visual and audio information based on the results.

[1226] Example: The server recognizes the user's emotions in real time and incorporates that feedback into the generative model.

[1227] NFT conversion

[1228] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[1229] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[1230] Terminal

[1231] Entering and Submitting Data

[1232] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[1233] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[1234] Receiving and displaying the generated results

[1235] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[1236] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[1237] Feedback collection with emotion engine

[1238] The terminal transmits the emotions of the user while viewing the generated content to the emotion engine in real time.

[1239] Example: The device analyzes the user's facial expressions and speech while watching, and sends the emotional data to the server's emotion engine.

[1240] User

[1241] Talent selection and data provision

[1242] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[1243] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[1244] Check the generated results and request corrections

[1245] The user can review the visual and audio information of the generated talent, request corrections from the server if necessary, and provide feedback using the emotion engine to optimize the generation process.

[1246] Example: A marketer reviews the generated video and requests the server to modify specific facial expressions or lines, and the emotion engine learns from user feedback.

[1247] final use

[1248] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[1249] Example: A marketer distributes the final video for an advertising campaign.

[1250] The system of the present invention makes it possible to execute a high-quality, consistent advertising strategy that is in line with user emotions while perpetually maintaining the market value of a talent even after the talent has passed its prime.

[1251] The processing flow will be explained below.

[1252] Program processing flow

[1253] server

[1254] Step 1:

[1255] The server receives visual, audio, and textual information uploaded by users, including celebrity images, video files, audio files, and social media posts.

[1256] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[1257] Step 2:

[1258] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[1259] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[1260] Step 3:

[1261] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[1262] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[1263] Step 4:

[1264] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[1265] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[1266] Step 5:

[1267] The server uses the preprocessed data to train a generative model, integrating visual, audio, and text information to build a generative model that can recreate the talent in their prime.

[1268] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[1269] Step 6:

[1270] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[1271] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[1272] Step 7:

[1273] The server uses an emotion engine to analyze the user's emotions and reflects them as feedback to the generative model, thereby improving the quality of the generated content.

[1274] Example: The server collects data on the user's emotions and reflects it in the generation process.

[1275] Step 8:

[1276] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[1277] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[1278] Terminal

[1279] Step 1:

[1280] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[1281] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[1282] Step 2:

[1283] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[1284] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[1285] Step 3:

[1286] The device transmits emotional data in real time to a server while the user is viewing the generated content, and emotion recognition software is used to analyze emotions such as facial expressions and speech.

[1287] Example: The device analyzes the user's facial expressions and speech while watching and sends the emotional data to the server.

[1288] User

[1289] Step 1:

[1290] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[1291] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[1292] Step 2:

[1293] The user can review the visual and audio information of the generated talent, request corrections from the server if necessary, and provide feedback using the emotion engine to optimize the generation process.

[1294] Example: A marketer reviews the generated video and requests the server to modify specific facial expressions or lines, and the emotion engine learns from user feedback.

[1295] Step 3:

[1296] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[1297] Example: A marketer distributes the final video for an advertising campaign.

[1298] The above is a specific processing flow of the present invention. This system makes it possible to execute a high-quality, consistent advertising strategy that is in tune with the user's emotions while perpetually maintaining the market value of a talent even after they have passed their prime.

[1299] Example 2

[1300] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1301] In today's digital media environment, there is a need to generate high-quality, consistent visual and audio content while reusing the assets of past talent. However, current technologies lack a method for consistently processing visual, audio, and text information and training effective generative models, resulting in inconsistent quality of generated content. Furthermore, the lack of a function for adjusting content based on user emotions makes it difficult to provide an optimal entertainment experience.

[1302] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for extracting facial features from the visual information of the preprocessed information, means for denoising the preprocessed audio information, means for tokenizing and analyzing the preprocessed text information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for adjusting the generated visual and audio information based on user emotional feedback, and means for storing the generated information as an NFT. This enables effective reuse of past talent assets and the generation of high-quality, consistent visual and audio content adjusted based on user emotions.

[1303] "Visual information" is information that is presented in a form that can be seen with the eyes, such as an image or video file.

[1304] "Audio information" refers to information that is expressed in a form that can be heard through the ears, such as audio files or recorded data.

[1305] "Text information" refers to information that is written in a readable form, such as written characters or sentences.

[1306] "Preprocessing" refers to the process of preparing collected raw data in a form that is easier to analyze, and includes processes such as noise removal and tokenization.

[1307] "Facial features" refers to information extracted from visual information about a person's facial shape and characteristic points such as the eyes, nose, and mouth.

[1308] "Noise reduction" is a process for removing unnecessary background sounds and interference sounds contained in audio information.

[1309] "Tokenization" is a process of dividing text information into small units such as words and phrases and analyzing their meanings and relationships.

[1310] A "generative model" is an algorithm used to generate new visual or audio information based on input data, and includes GANs and Transformers.

[1311] A "trained generative model" is a generative algorithm that has been optimized using preprocessed data and is used to generate new content based on a specified scenario.

[1312] "Emotional feedback" is a process of collecting evaluations and opinions based on the user's emotional state and adjusting the generated content based on that information.

[1313] An "NFT" is a token that uses blockchain technology to prove ownership of digital data and represents a one-of-a-kind digital asset.

[1314] This system generates high-quality visual and audio information by integrating visual, audio, and text information, preprocessing the data, and training a generative model. The generated information is stored as NFTs, which retain their value over the long term. This system provides optimized content based on specific scenarios and user emotional feedback.

[1315] server

[1316] Data collection

[1317] The server receives visual, audio, and text information uploaded by the user through the device. Specifically, it receives HTTP requests and saves image files and text data in storage.

[1318] Example: A server receives and stores actor film clips, interview audio files, and social media posts sent by a marketer.

[1319] Data Preprocessing

[1320] The server preprocesses the collected visual information and extracts facial features using a facial recognition algorithm such as FaceNet. It also preprocesses the audio information by removing noise using FFmpeg or Audacity. It then tokenizes and analyzes the text information using natural language processing libraries such as NLTK and spaCy.

[1321] Examples: Extracting actors' facial features from movie clips, removing background noise from interview audio, and tokenizing and analyzing social media posts.

[1322] Training a generative model

[1323] The server uses the preprocessed data to train a generative model such as a GAN or Transformer, using TensorFlow or PyTorch to integrate facial features, audio data, and text data to optimize the model.

[1324] Example: Training a generative model of AI talent using facial features, voice features, and thought patterns derived from text data.

[1325] AI Talent Generation

[1326] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a given scenario, inputs a prompt (e.g., "Introducing a new product") into the model, and stores the generated video and audio.

[1327] Example: Based on a specific product promotion scenario, generate video and audio of an actor introducing a product.

[1328] Utilizing the Emotion Engine

[1329] The server uses an emotion engine to analyze user feedback and adjust the generated content. The emotion engine recognizes user emotions in real time and fine-tunes the generative model based on them.

[1330] Example: Collecting the user's emotions as feedback and adjusting the actor's facial expressions based on that data.

[1331] NFT conversion

[1332] The server stores the generated talent's visual and audio information as an NFT, which is recorded on the Ethereum blockchain with a digital signature, making it a unique digital asset.

[1333] Example: The generated video and audio of an actor is registered as an NFT, and its ownership information is recorded on the blockchain.

[1334] Terminal

[1335] Entering and Submitting Data

[1336] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits it to the server.

[1337] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[1338] Receiving and displaying the generated results

[1339] The terminal receives the generated content from the server and displays it to the user, allowing the user to review the results and make correction requests if necessary.

[1340] Example: Display a video of an actor introducing a new product sent from the server and play an audio file.

[1341] Feedback collection with emotion engine

[1342] While the user is viewing the generated content, the device analyzes facial expressions and speech in real time and sends the results to the emotion engine.

[1343] Example: Analyze the facial expressions and speech of the user while watching, and send the emotional data to the emotion engine on the server.

[1344] User

[1345] Talent selection and data provision

[1346] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[1347] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[1348] Check the generated results and request corrections

[1349] The user can review the generated visual and audio information, request corrections from the server, and even provide feedback using the emotion engine to optimize the generation process.

[1350] Example: The generated video is reviewed, and specific facial expressions or lines are corrected by the server, and the emotion engine learns from the user's feedback.

[1351] final use

[1352] The user ultimately selects the content they are satisfied with and uses it for advertising campaigns and promotions.

[1353] Example: A marketer distributes the final video for an advertising campaign.

[1354] Prompt Sentence Examples

[1355] "Generate a new product promotional video using the face and voice of a famous actor from the 1980s. Use uploaded film clips and interview audio."

[1356] "Please use the user's emotional data as a reference to optimize the facial expressions of the actors in the generated video to make them appear happier."

[1357] The above is an embodiment of the present invention.

[1358] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1359] Step 1: Data collection

[1360] The server receives visual, audio, and text information uploaded by users through their devices. Specifically, it receives HTTP requests and saves image files and text data in storage. The input to this process is the data provided by the user, and the output is the saved raw data. Specifically, the server receives and saves actor film clips, interview audio files, and social media posts sent by marketers.

[1361] Step 2: Data preprocessing (visual information)

[1362] The server preprocesses the received visual information and extracts facial features using a facial recognition algorithm such as FaceNet. The input to this process is the collected visual information, and the output is the extracted facial feature information. Specifically, the server uses OpenCV to analyze frames from a movie clip, detect facial regions, and extract feature points (eyes, nose, mouth, etc.).

[1363] Step 3: Data preprocessing (audio information)

[1364] The server uses FFmpeg and Audacity to preprocess the audio information and remove noise. The input of this process is the collected audio information, and the output is clear audio data with noise removed. Specifically, the server uses FFmpeg to read the audio file, apply a filter to remove noise, and generate clear audio data.

[1365] Step 4: Data preprocessing (text information)

[1366] The server preprocesses the text information and performs tokenization using a natural language processing library (NLTK or spaCy). The input to this process is the collected text information, and the output is tokenized text data. Specifically, the server reads the text data, splits it into tokens, extracts keywords and important phrases, and stores them in a database.

[1367] Step 5: Training the generative model

[1368] The server uses the preprocessed data to train a generative model such as a GAN or Transformer. The input for this process is the preprocessed visual, audio, and text information, and the output is a trained generative model. Specifically, the server uses TensorFlow or PyTorch to train the model using facial features, audio data, and text data as input, and saves the trained model.

[1369] Step 6: Generate AI talent

[1370] The server uses a trained generative model to generate visual and audio information for an AI talent based on a specified scenario. The input to this process is the scenario information and the trained generative model, and the output is the generated visual and audio information. Specifically, the server inputs a prompt sentence (e.g., "Introducing a new product") into the model and saves the generated video and audio.

[1371] Step 7: Leverage your emotional engine

[1372] The device analyzes the user's facial expressions and speech in real time while watching the generated content and sends the results to the emotion engine. The input for this process is the user's facial expression data and voice data, and the output is emotional feedback. Specifically, the device's camera and microphone are used to collect the user's facial expression data and voice data, and the data is sent to the emotion engine in real time.

[1373] The server uses an emotion engine to analyze user feedback and adjusts the generated content by incorporating the results into the generative model. The input to this process is the emotional feedback sent from the device, and the output is the adjusted generated content. Specifically, the emotion engine identifies the user's emotions and optimizes the generated video and audio based on them.

[1374] Step 8: NFT

[1375] The server stores the generated visual and audio information of the talent as an NFT. The input of this process is the generated visual and audio information, and the output is an NFT recorded on the blockchain. Specifically, the server digitally signs the generated data and records it on a blockchain network such as Ethereum.

[1376] (Application example 2)

[1377] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1378] In the prior art, there was no effective method for generating visual and audio information to recreate a specific celebrity's peak performance, and it was difficult to reflect user emotional feedback in real time. Furthermore, there was insufficient protection of the value of the generated content and proof of ownership, making it difficult to continue providing high-quality content that satisfies users. The present invention aims to solve these problems.

[1379] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for storing the generated information as a non-fungible token, and means for collecting and analyzing user feedback using an emotion engine to allow users to customize the generated visual and audio information and reflecting the feedback in the generation process. This makes it possible to continue providing high-quality, customizable content and also makes it easier to preserve the value of the generated content and prove ownership.

[1380] "Visual information" is a general term for image and video data collected from users and other sources.

[1381] "Audio Information" refers collectively to audio data collected from users and other sources.

[1382] "Text Information" refers collectively to written or typed character information collected from users or other sources.

[1383] "Preprocessing" is the process of formatting collected visual, audio, and text information to make it easier to analyze.

[1384] A "generative model" is a machine learning or deep learning algorithm that uses collected and preprocessed data to generate new visual or audio information.

[1385] A "non-fungible token" is a digital asset with unique value based on a blockchain, typically used for proof of ownership and store of value purposes.

[1386] An "emotion engine" is an algorithm and system that analyzes users' emotions in real time and collects and reflects the results as feedback.

[1387] "User feedback" refers to the user's impressions and reactions to the generated visual and audio information, and requests for modifications based on the impressions and reactions.

[1388] This invention provides a system that collects visual, audio, and text information, preprocesses them, and then generates visual and audio information using a generative model. The system includes a means for storing the generated information as a non-fungible token (NFT). The system also uses an emotion engine to allow users to customize the generated visual and audio information, collects and analyzes user feedback, and incorporates that feedback into the generation process.

[1389] System configuration and operation overview

[1390] 1. Visual information collection and preprocessing:

[1391] The server receives image and video files uploaded by users, such as old movie clips or interview footage.

[1392] Use OpenCV to resize and reshape the image and extract facial features.

[1393] 2. Acquisition and preprocessing of audio information:

[1394] The server receives audio files uploaded by users, for example, interview audio data.

[1395] Use pydub to convert the audio data to mono and format it in a way that is suitable for the speech recognition model.

[1396] 3. Text information collection and preprocessing:

[1397] The server receives text information uploaded by the user, such as a social media post or a script.

[1398] Text information is tokenized and converted into a format that can be input into natural language processing models such as GPT-2.

[1399] 4. Training the generative model and generating visual and audio information:

[1400] The server trains a generative model (e.g., GAN, Transformer) using the preprocessed data.

[1401] A trained generative model is used to generate visual and audio information.

[1402] 5. Feedback collection and analysis using emotion engine:

[1403] The terminal presents the generated visual and audio information to the user and collects the user's emotional feedback in real time.

[1404] The emotion engine analyzes the user's facial expressions and voice and sends the feedback to the server.

[1405] 6. Optimization of generated information and NFTization:

[1406] The server optimizes the generation process based on user feedback.

[1407] The final content generated is stored as a non-fungible token (NFT) to preserve value and prove ownership.

[1408] Specific examples

[1409] If an advertising agency wants to use a famous celebrity from the 1990s to promote a new product, they can use this system. They can upload old movie clips, interview audio, social media posts, etc. to train a generative model. The system then generates an advertising video, and users can provide real-time feedback via smartphone or head-mounted display. The final optimized advertising video is stored as an NFT, preserving ownership and value.

[1410] Example prompt sentence:

[1411] "Generate an advertising video for new product X featuring famous 90s actor A. Use the following information: movie clips, interview audio, and social media posts."

[1412] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1413] Step 1:

[1414] The server collects visual information from users. Specifically, users upload image or video files, such as old movie clips or interview footage. The server receives these files, resizes and reshapes the images using OpenCV, and extracts facial features.

[1415] Input: User-uploaded film clips and interview footage

[1416] Processing: Resize, shape, and extract facial features using OpenCV

[1417] Output: Preprocessed visual data

[1418] Step 2:

[1419] The server collects audio information from users, specifically, by uploading interview recordings or other audio files. The server receives these audio files and uses pydub to convert them to mono and remove background noise.

[1420] Input: Audio file uploaded by the user

[1421] Processing: Mono conversion and noise removal using pydub

[1422] Output: Preprocessed audio data

[1423] Step 3:

[1424] The server collects text information from users. Specifically, users upload text data such as social media posts and scripts. The server receives this text information, tokenizes it, and converts it into a format suitable for input into a natural language processing model (such as GPT-2).

[1425] Input: Text data uploaded by the user

[1426] Processing content: Tokenization, format conversion

[1427] Output: Preprocessed text data

[1428] Step 4:

[1429] The server uses the preprocessed visual, audio, and text data to train a generative model, which is trained using machine learning algorithms such as GAN and Transformer to generate visual and audio information to recreate the talent in their prime.

[1430] Input: Preprocessed visual data, audio data, and text data

[1431] Processing content: Training of generative models (GAN and Transformer)

[1432] Output: Generated visual and audio information

[1433] Step 5:

[1434] The server transmits the generated visual and audio information to the user's device, which then presents the information to the user. The user can then view the generated visual and audio information using a smartphone or head-mounted display and provide emotional feedback.

[1435] Input: Generated visual and audio information

[1436] Processing: Presenting information and collecting emotional feedback

[1437] Output: User's emotional feedback

[1438] Step 6:

[1439] The emotion engine analyzes user feedback, including facial expressions and tone of voice, and optimizes the generated visual and audio information based on the results. Feedback includes the user's facial expressions and tone of voice, and by analyzing these, it adjusts the output of the generative model.

[1440] Input: User's emotional feedback

[1441] Processing content: Analysis of emotional feedback and optimization of generated information

[1442] Output: Optimized visual and audio information

[1443] Step 7:

[1444] The server stores the optimized visual and audio information as non-fungible tokens (NFTs), a process that involves creating digital signatures and recording them on the blockchain.

[1445] Input: Optimized visual and audio information

[1446] Processing: Save as NFT, create digital signature, record on blockchain

[1447] Output: Visual and audio information stored as NFTs

[1448] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1449] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1450] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1451] [Fourth embodiment]

[1452] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1453] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1454] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1455] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1456] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1457] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1458] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1459] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1460] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1461] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1462] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1463] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1464] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1465] The system of the present invention collects visual, audio, and textual information, preprocesses the data, and then trains a generative model to generate visual and audio information that recreates the talent's appearance in their prime. The generated information is stored as an NFT, which can retain its value for a long period of time.

[1466] server

[1467] Data collection

[1468] The server receives visual, audio, and textual information uploaded by users, including talent images, video files, audio files, and social media posts.

[1469] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[1470] Data Preprocessing

[1471] The server analyzes the facial features of the received visual information, denoises the audio information, and tokenizes the text information.

[1472] Example: A server extracts facial features from a movie clip, removes noise from an audio file, or tokenizes and extracts features from a social media post.

[1473] Training a generative model

[1474] The server uses the preprocessed data to train a generative model (e.g., GAN or Transformer), which is used to integrate and reproduce the talent's visual, vocal, and thought patterns.

[1475] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[1476] AI Talent Generation

[1477] The server uses the trained generative model to generate visual and audio information for the talent based on the specified scenario.

[1478] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[1479] NFT conversion

[1480] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[1481] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[1482] Terminal

[1483] Entering and Submitting Data

[1484] The terminal checks for errors in the visual, audio, and text information entered by the user and sends the information to the server.

[1485] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[1486] Receiving and displaying the generated results

[1487] The terminal receives the generated visual and audio information from the server and displays it to the user.

[1488] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[1489] User

[1490] Talent selection and data provision

[1491] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[1492] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[1493] Check the generated results and request corrections

[1494] The user reviews the generated talent's visual and audio information and requests modifications from the server if necessary.

[1495] Example: A marketing person reviews the generated video and requests the server to correct specific facial expressions or lines.

[1496] final use

[1497] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[1498] Example: A marketer distributes the final video for an advertising campaign.

[1499] The system of the present invention makes it possible to execute a high-quality and consistent advertising strategy while perpetually maintaining the market value of a talent even after the talent has passed its prime.

[1500] The processing flow will be explained below.

[1501] Program processing flow

[1502] server

[1503] Step 1:

[1504] The server receives visual, audio, and textual information uploaded by users, including celebrity images, video files, audio files, and social media posts.

[1505] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[1506] Step 2:

[1507] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[1508] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[1509] Step 3:

[1510] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[1511] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[1512] Step 4:

[1513] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[1514] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[1515] Step 5:

[1516] The server uses the preprocessed data to train a generative model, integrating visual, audio, and text information to build a generative model that can recreate the talent in their prime.

[1517] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[1518] Step 6:

[1519] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[1520] Example: A server generates video and audio of an actor introducing a new product based on a specific scenario.

[1521] Step 7:

[1522] The server creates a digital signature and records it on the blockchain to store the generated visual and audio information as an NFT.

[1523] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[1524] Terminal

[1525] Step 1:

[1526] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[1527] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[1528] Step 2:

[1529] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[1530] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[1531] User

[1532] Step 1:

[1533] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[1534] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[1535] Step 2:

[1536] The user can check the generated visual and audio information of the talent and request corrections from the server as necessary, thereby creating optimal advertising materials.

[1537] Example: A marketing person reviews the generated video and requests the server to correct specific facial expressions or lines.

[1538] Step 3:

[1539] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[1540] Example: A marketer distributes the final video for an advertising campaign.

[1541] The above is a specific processing flow of the present invention, which makes it possible to execute a high-quality and consistent advertising strategy while perpetually maintaining the market value of a talent even after they have passed their prime.

[1542] Example 1

[1543] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1544] Conventional digital talent reproduction technologies rely on a single source of information (such as visual information only or audio information only), making it difficult to reproduce a talent's peak performance. Furthermore, there is no way to preserve the value of the generated digital data over the long term. This makes it difficult to implement high-quality, consistent advertising strategies.

[1545] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1546] In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative AI model using the preprocessed information, means for generating visual and audio information using the trained generative AI model, and means for digitally signing the generated visual and audio information and recording it on a blockchain as a non-fungible token. This makes it possible to reproduce a talent's peak years with high accuracy and preserve the value of the generated digital data for a long period of time.

[1547] "Visual information" refers to information that can be seen and recognized with the eyes, such as images and videos.

[1548] "Audio information" refers to information that can be heard by the ears, such as audio data or recording files.

[1549] "Text information" refers to information such as sentences and character data, the contents of which can be understood by reading it.

[1550] "Preprocessing" refers to a series of processes that convert collected information into a form suitable for input into a generative model, such as parsing, formatting, and tokenization.

[1551] "Generative AI model" refers to an artificial intelligence algorithm that is trained using collected and preprocessed data to generate new data (visual and audio information) based on specific inputs.

[1552] A "non-fungible token (NFT)" refers to a unique digital token that can prove ownership of a digital asset using blockchain technology.

[1553] "Digital Signature" refers to a cryptographic signature created to prove that digital data has not been tampered with.

[1554] "Blockchain" refers to a technology that maintains a series of records of digital transactions and manages them in a distributed manner across multiple computers.

[1555] The system of the present invention collects visual, audio, and text information, preprocesses the data, and then trains a generative AI model to generate visual and audio information that recreates the talent's appearance in their prime. The generated information is stored as a non-fungible token (NFT), which can retain its value for a long period of time.

[1556] server

[1557] The server uses HTTP requests to receive visual, audio, and textual information uploaded by users, such as images, video files, audio files, social media posts, etc. The server then validates the data and stores it in a database.

[1558] When preprocessing visual information, we use libraries such as OpenCV and Dlib to analyze facial features. When preprocessing audio information, we use audio processing libraries such as LibROSA to remove noise. When preprocessing text information, we use natural language processing libraries such as SpaCy and NLTK to tokenize and extract features.

[1559] TensorFlow and PyTorch are used to train generative AI models (e.g., GANs and Transformers) using the preprocessed data, which then builds highly accurate models that integrate the talent's appearance, voice, and thought patterns.

[1560] Based on the trained generative AI model, the server generates visual and audio information according to prompts given by the client. For example, a prompt such as "Generate a video in which actor X from the 1990s introduces new product Y. Product Y's features are A and B, and actor X should introduce the product in an enthusiastic and friendly tone."

[1561] The generated visual and audio information is digitally signed and recorded on the blockchain as a non-fungible token (NFT), using blockchain technologies such as Ethereum and Flow.

[1562] Terminal

[1563] The device checks the visual, audio, and text information entered by the user for errors and sends it to the server in the appropriate format, for example, checking the format of a movie clip, checking the sampling rate of an interview audio, and ensuring that a social media post is properly formatted.

[1564] It receives generated visual and audio information from the server, analyzes the generated results, and displays them on the screen or plays them back to the user. The user can review the generated content and request corrections if necessary.

[1565] User

[1566] Users log in to the system, select a period of time they want to focus on for a talent, and upload visual, audio, and text information related to that period. For example, a marketer could select an actor's data from the 1990s and upload film clips, audio interviews, and social media posts from that period.

[1567] Users can check the visual and audio information of the generated talent and request the server to modify specific facial expressions, lines, etc. Finally, they can select the content they are satisfied with and use it in corporate advertising campaigns and promotions, enabling them to execute high-quality, consistent advertising strategies while accurately recreating the talent in their prime.

[1568] This system allows users to easily create, store, and use digital information about celebrities, even after they have passed their prime, while maintaining their market value in perpetuity.

[1569] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1570] Step 1: Enter and submit data

[1571] Subject: Terminal

[1572] Specific operation: The user inputs visual, audio, and text information through the terminal. The terminal checks for errors in this data and converts it into the required format. Specifically, it checks the format of the movie clip and the audio sampling rate.

[1573] Input: User-uploaded film clips, interview audio files, social media posts

[1574] Output: Error checked and formatted data

[1575] Data processing: The device checks the format and data integrity and sends it to the server via an HTTP request.

[1576] Step 2: Receiving and storing data

[1577] Subject: Server

[1578] What happens: The server receives the HTTP request, checks the file format, verifies that it is in the correct format, and saves it to the database.

[1579] Input: Error-checked and formatted data sent from the terminal

[1580] Output: Visual, audio and text information stored in a database

[1581] Data processing: The server checks the data format and registers it in the database in a consistent state.

[1582] Step 3: Preprocessing the data

[1583] Subject: Server

[1584] Specific operation: The server uses libraries such as OpenCV and Dlib to extract facial features from visual information, audio processing libraries such as LibROSA to remove noise from audio data, and text information is tokenized and features are extracted using SpaCy and NLTK.

[1585] Input: Visual, audio, and text information stored in a database

[1586] Output: Preprocessed visual, audio, and text information

[1587] Data processing: The server analyzes visual information, reduces audio noise, tokenizes text, and extracts features.

[1588] Step 4: Training the generative AI model

[1589] Subject: Server

[1590] How it works: The server uses the preprocessed data to train a generative AI model (such as a GAN or Transformer). The training process is performed using TensorFlow or PyTorch to improve the model's accuracy.

[1591] Input: Preprocessed visual, audio, and text information

[1592] Output: A trained generative AI model

[1593] Data Computation: The server processes the data in batches and runs a training loop to improve the model's performance. It monitors the model's accuracy and loss function and adjusts hyperparameters as needed.

[1594] Step 5: Generate AI talent

[1595] Subject: Server

[1596] How it works: The server uses a trained generative AI model to generate visual and audio information based on prompts provided by the client.

[1597] Input: prompt, trained generative AI model

[1598] Output: Generated visual and audio information

[1599] Data calculation: The server analyzes the given prompt and inputs it into the generative AI model, which generates visual and audio information and retrieves that data.

[1600] Step 6: NFT

[1601] Subject: Server

[1602] How it works: The server digitally signs the generated visual and audio information and records it on the blockchain as a non-fungible token (NFT). It uses blockchain technologies such as Ethereum and Flow.

[1603] Input: Generated visual and audio information

[1604] Output: NFT recorded on the blockchain

[1605] Data processing: The server creates a digital signature and records ownership information on the blockchain.

[1606] Step 7: Receive and display the generated results

[1607] Subject: Terminal

[1608] Specific operation: The device receives the visual and audio information generated by the server and displays it to the user. The generated results are analyzed and displayed on the screen or played back as audio.

[1609] Input: Generated visual and audio information sent from the server

[1610] Output: Visual and audio information displayed to the user

[1611] Data processing: The device analyzes the generated data and displays and plays it appropriately for the user.

[1612] Step 8: Check the results and request corrections

[1613] Subject: User

[1614] Specific operation: The user checks the generated visual and audio information and requests the server to correct specific facial expressions, lines, etc.

[1615] Input: Generated visual and audio information

[1616] Output: Correction request

[1617] Data processing: The user checks the generated results, clarifies any corrections, and sends a request to the server.

[1618] Through these processing steps, users can create high-quality digital representations of their talent and use the generated information in advertising campaigns and promotions.

[1619] (Application example 1)

[1620] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1621] Conventional methods have had the problem that once a celebrity's prime passes, their market value declines, limiting their use in advertising and promotions. Furthermore, it is difficult to generate high-quality visual and audio information, and there is no way to maintain its value over the long term. Furthermore, when applied to virtual stores, there is a lack of means to enable real-time interactive customer service.

[1622] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1623] In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for recording the generated information on a blockchain and creating a digital signature for maintaining its value over the long term, and means for reproducing the generated visual and audio information in a virtual environment based on a user-specified scenario and displaying it to the user. This makes it possible to provide interactive customer service in a virtual store while perpetually maintaining the market value of a talent even after they have passed their prime.

[1624] "Visual information" refers to images, video clips, and other visual data.

[1625] "Audio information" refers to sound files, recordings, and other audio data.

[1626] "Text information" refers to sentences, notes, text files, and other text data.

[1627] "Preprocessing" refers to the process of analyzing collected data, removing noise, and extracting features.

[1628] "Generative model" refers to a trained algorithm for generating visual and audio information using artificial intelligence techniques.

[1629] "Digital signature" refers to a cryptographic technique used to ensure the authenticity and authenticity of generated information.

[1630] "Blockchain" refers to a system that links information in a chain and uses distributed ledger technology to prevent data tampering.

[1631] A "virtual environment" refers to a virtual space or situation generated by computer simulation.

[1632] This invention is a system for realizing interactive customer service in a virtual environment, and is composed of three elements: a server, a terminal, and a user.

[1633] server

[1634] The server collects visual, audio, and text information and preprocesses it. Specifically, the following processes are performed:

[1635] 1. Data Collection

[1636] The server receives visual information (e.g., images, video clips), audio information (e.g., audio recordings), and text information (e.g., social media posts) uploaded by users.

[1637] 2. Data Preprocessing

[1638] The received visual information is analyzed for facial features using an image processing library (e.g., OpenCV).

[1639] For audio information, an audio processing library is used to remove noise.

[1640] For text information, tokenization is performed using a natural language processing library (e.g., Transformers).

[1641] 3. Training the generative model

[1642] The pre-processed data is used to train a generative AI model (e.g., GAN or Transformer) that integrates the talent's visual, vocal, and thought patterns.

[1643] 4. AI Talent Generation

[1644] A trained generative model is used to generate visual and audio information based on a user-specified scenario.

[1645] 5. NFT conversion

[1646] The generated visual and audio information is recorded on the blockchain, a digital signature is created and stored as an NFT.

[1647] Terminal

[1648] The terminal has the following features:

[1649] 1. Enter and submit data

[1650] The visual, audio, and text information entered by the user is checked for errors and then sent to the server.

[1651] 2. Receiving and displaying the generated results

[1652] The system receives visual and audio information generated by the server and displays it to the user. Specifically, it plays video and audio of an AI talent introducing products in a virtual store environment.

[1653] User

[1654] The user performs the following operations:

[1655] 1. Talent selection and data provision

[1656] Users log in, select a talent's prime period, and upload visual, audio, and text information related to that period.

[1657] 2. Check the generated results and request corrections

[1658] Check the visual and audio information of the generated AI talent and request corrections from the server if necessary.

[1659] 3. End use

[1660] Finally, the content that satisfies the customer is selected and used for interactive customer service in the virtual store.

[1661] Specific examples

[1662] For example, a marketer could collect data on popular actors from the 1990s and generate a scenario to introduce a new product in a virtual store.

[1663] "Generate a scenario in which a popular actor from the 1990s appears in a virtual store and gives a detailed explanation of new products."

[1664] This system enables interactive customer service in a virtual store while perpetually maintaining the market value of talent even after they have passed their prime. The hardware used is a server and end-user devices, and the specific software includes OpenCV, Transformers, audio processing libraries, blockchain technology, etc.

[1665] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1666] Step 1:

[1667] The server collects visual, audio, and text information, including images, video clips, audio recordings, and social media posts uploaded by users. Specific data formats used include JPEG, MP4, MP3, and TXT. The received data is then temporarily stored in the server's database.

[1668] Step 2:

[1669] The server preprocesses the collected visual, audio, and text information. Specifically, for visual information, OpenCV is used to analyze facial features and a facial recognition algorithm is used to extract facial landmarks (eyes, nose, mouth, etc.). For audio information, an audio processing library is used to remove noise and convert it into clear audio data. For text information, the Transformers library is used to tokenize the data and convert each word or phrase into a vector representation. This preprocessed data is compiled into a training dataset and used in the next stage.

[1670] Step 3:

[1671] The server uses the preprocessed data to train a generative model. Specifically, the training dataset is input into a GAN (generative adversarial network) or Transformer model, and the model parameters are optimized. This results in a generative AI model capable of integrating and reproducing the talent's visual, vocal, and thought patterns. Model training is performed efficiently using a GPU to increase computational speed.

[1672] Step 4:

[1673] The server generates visual and audio information using a trained generative model. Based on a scenario specified by the user, a prompt sentence, such as "A celebrity will introduce a new product in a virtual store," is input into the generative model. Based on this prompt sentence, the model generates the celebrity's visual information (e.g., facial and body movements) and audio information (e.g., audio explaining the product). The generated data is temporarily stored in the server's storage.

[1674] Step 5:

[1675] The server stores the generated visual and audio information as an NFT. Specifically, it creates a digital signature for the generated data and records the information using blockchain technology. This guarantees the uniqueness of the generated data, making it possible to maintain its reliability and authenticity over the long term.

[1676] Step 6:

[1677] The device receives and displays the generated results. Specifically, it receives the generated visual and audio information sent from the server and displays and plays it for the user. For example, it plays video and audio of an AI talent introducing a product in a simulated virtual store environment. Based on this, the user can understand the details of the product and consider purchasing it.

[1678] Step 7:

[1679] The user checks the generated results and requests corrections. Specifically, the user requests the server to make minor corrections to the generated video and audio (e.g., changing specific facial expressions or lines). The server receives this request, makes corrections using the generative model again, and provides a satisfactory result.

[1680] Step 8:

[1681] Users can then select the content they are most satisfied with and use it in the virtual store. The generated AI talent's video and audio can then be used in actual promotions and advertising campaigns. This provides users with an interactive customer service experience and effectively communicates the appeal of the products.

[1682] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1683] The system of the present invention collects visual, audio, and text information, preprocesses the data, and then trains a generative model to generate visual and audio information that recreates the celebrity's appearance in their prime. Furthermore, by incorporating an emotion engine that recognizes user emotions and incorporating user feedback into the generation process, more optimal content generation is achieved. Furthermore, the generated information is stored as an NFT, which can maintain its value for a long period of time.

[1684] server

[1685] Data collection

[1686] The server receives visual, audio, and textual information uploaded by users, including talent images, video files, audio files, and social media posts.

[1687] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[1688] Data Preprocessing

[1689] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[1690] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[1691] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[1692] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[1693] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[1694] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[1695] Training a generative model

[1696] The server uses the preprocessed data to train a generative model (e.g., GAN or Transformer), which is used to integrate and reproduce the talent's visual, vocal, and thought patterns.

[1697] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[1698] AI Talent Generation

[1699] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[1700] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[1701] Utilizing the Emotion Engine

[1702] The server adjusts the output of the generative model using an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's feedback and optimizes the generated visual and audio information based on the results.

[1703] Example: The server recognizes the user's emotions in real time and incorporates that feedback into the generative model.

[1704] NFT conversion

[1705] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[1706] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[1707] Terminal

[1708] Entering and Submitting Data

[1709] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[1710] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[1711] Receiving and displaying the generated results

[1712] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[1713] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[1714] Feedback collection with emotion engine

[1715] The terminal transmits the emotions of the user while viewing the generated content to the emotion engine in real time.

[1716] Example: The device analyzes the user's facial expressions and speech while watching, and sends the emotional data to the server's emotion engine.

[1717] User

[1718] Talent selection and data provision

[1719] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[1720] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[1721] Check the generated results and request corrections

[1722] The user can review the visual and audio information of the generated talent, request corrections from the server if necessary, and provide feedback using the emotion engine to optimize the generation process.

[1723] Example: A marketer reviews the generated video and requests the server to modify specific facial expressions or lines, and the emotion engine learns from user feedback.

[1724] final use

[1725] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[1726] Example: A marketer distributes the final video for an advertising campaign.

[1727] The system of the present invention makes it possible to execute a high-quality, consistent advertising strategy that is in line with user emotions while perpetually maintaining the market value of a talent even after the talent has passed its prime.

[1728] The processing flow will be explained below.

[1729] Program processing flow

[1730] server

[1731] Step 1:

[1732] The server receives visual, audio, and textual information uploaded by users, including celebrity images, video files, audio files, and social media posts.

[1733] Example: A server receives an actor's film clip, an interview audio file, or a social media post.

[1734] Step 2:

[1735] The server pre-processes the received visual information and extracts facial features using a facial recognition algorithm.

[1736] Example: A server extracts facial features of actors from movie clips and stores them in a database.

[1737] Step 3:

[1738] The server pre-processes the audio information and removes noise, using audio processing algorithms to remove background noise and obtain clear audio data.

[1739] Example: A server removes background noise from an interview audio and extracts only the actor's voice clearly.

[1740] Step 4:

[1741] The server pre-processes and tokenizes the text information, then uses natural language processing techniques to analyze the text data and extract keywords and important phrases.

[1742] Example: A server tokenizes social media posts and obtains data to analyze the actor's thought patterns.

[1743] Step 5:

[1744] The server uses the preprocessed data to train a generative model, integrating visual, audio, and text information to build a generative model that can recreate the talent in their prime.

[1745] Example: The server trains a generative model by integrating an actor's facial features, vocal features, and thought patterns extracted from text.

[1746] Step 6:

[1747] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a specified scenario.

[1748] Example: A server generates video and audio of an actor introducing a product based on a specific product introduction scenario.

[1749] Step 7:

[1750] The server uses an emotion engine to analyze the user's emotions and reflects them as feedback to the generative model, thereby improving the quality of the generated content.

[1751] Example: The server collects data on the user's emotions and reflects it in the generation process.

[1752] Step 8:

[1753] The server creates a digital signature and records it on the blockchain to store the generated talent's visual and audio information as an NFT.

[1754] Example: The server registers the generated actor's video and audio data as an NFT and records ownership information on the blockchain.

[1755] Terminal

[1756] Step 1:

[1757] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits this information to the server.

[1758] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[1759] Step 2:

[1760] The device receives visual and audio information of the generated AI talent from the server and displays it to the user, allowing the user to review the results and request corrections if necessary.

[1761] Example: A terminal displays a video of an actor introducing a new product sent from a server and plays an audio file.

[1762] Step 3:

[1763] The device transmits emotional data in real time to a server while the user is viewing the generated content, and emotion recognition software is used to analyze emotions such as facial expressions and speech.

[1764] Example: The device analyzes the user's facial expressions and speech while watching and sends the emotional data to the server.

[1765] User

[1766] Step 1:

[1767] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[1768] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[1769] Step 2:

[1770] The user can review the visual and audio information of the generated talent, request corrections from the server if necessary, and provide feedback using the emotion engine to optimize the generation process.

[1771] Example: A marketer reviews the generated video and requests the server to modify specific facial expressions or lines, and the emotion engine learns from user feedback.

[1772] Step 3:

[1773] Users ultimately select the content they are satisfied with and use it in the company's advertising campaigns and promotions.

[1774] Example: A marketer distributes the final video for an advertising campaign.

[1775] The above is a specific processing flow of the present invention. This system makes it possible to execute a high-quality, consistent advertising strategy that is in tune with the user's emotions while perpetually maintaining the market value of a talent even after they have passed their prime.

[1776] Example 2

[1777] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1778] In today's digital media environment, there is a need to generate high-quality, consistent visual and audio content while reusing the assets of past talent. However, current technologies lack a method for consistently processing visual, audio, and text information and training effective generative models, resulting in inconsistent quality of generated content. Furthermore, the lack of a function for adjusting content based on user emotions makes it difficult to provide an optimal entertainment experience.

[1779] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for extracting facial features from the visual information of the preprocessed information, means for denoising the preprocessed audio information, means for tokenizing and analyzing the preprocessed text information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for adjusting the generated visual and audio information based on user emotional feedback, and means for storing the generated information as an NFT. This enables effective reuse of past talent assets and the generation of high-quality, consistent visual and audio content adjusted based on user emotions.

[1780] "Visual information" is information that is presented in a form that can be seen with the eyes, such as an image or video file.

[1781] "Audio information" refers to information that is expressed in a form that can be heard through the ears, such as audio files or recorded data.

[1782] "Text information" refers to information that is written in a readable form, such as written characters or sentences.

[1783] "Preprocessing" refers to the process of preparing collected raw data in a form that is easier to analyze, and includes processes such as noise removal and tokenization.

[1784] "Facial features" refers to information extracted from visual information about a person's facial shape and characteristic points such as the eyes, nose, and mouth.

[1785] "Noise reduction" is a process for removing unnecessary background sounds and interference sounds contained in audio information.

[1786] "Tokenization" is a process of dividing text information into small units such as words and phrases and analyzing their meanings and relationships.

[1787] A "generative model" is an algorithm used to generate new visual or audio information based on input data, and includes GANs and Transformers.

[1788] A "trained generative model" is a generative algorithm that has been optimized using preprocessed data and is used to generate new content based on a specified scenario.

[1789] "Emotional feedback" is a process of collecting evaluations and opinions based on the user's emotional state and adjusting the generated content based on that information.

[1790] An "NFT" is a token that uses blockchain technology to prove ownership of digital data and represents a one-of-a-kind digital asset.

[1791] This system generates high-quality visual and audio information by integrating visual, audio, and text information, preprocessing the data, and training a generative model. The generated information is stored as NFTs, which retain their value over the long term. This system provides optimized content based on specific scenarios and user emotional feedback.

[1792] server

[1793] Data collection

[1794] The server receives visual, audio, and text information uploaded by the user through the device. Specifically, it receives HTTP requests and saves image files and text data in storage.

[1795] Example: A server receives and stores actor film clips, interview audio files, and social media posts sent by a marketer.

[1796] Data Preprocessing

[1797] The server preprocesses the collected visual information and extracts facial features using a facial recognition algorithm such as FaceNet. It also preprocesses the audio information by removing noise using FFmpeg or Audacity. It then tokenizes and analyzes the text information using natural language processing libraries such as NLTK and spaCy.

[1798] Examples: Extracting actors' facial features from movie clips, removing background noise from interview audio, and tokenizing and analyzing social media posts.

[1799] Training a generative model

[1800] The server uses the preprocessed data to train a generative model such as a GAN or Transformer, using TensorFlow or PyTorch to integrate facial features, audio data, and text data to optimize the model.

[1801] Example: Training a generative model of AI talent using facial features, voice features, and thought patterns derived from text data.

[1802] AI Talent Generation

[1803] The server uses a trained generative model to generate visual and audio information for the AI ​​talent based on a given scenario, inputs a prompt (e.g., "Introducing a new product") into the model, and stores the generated video and audio.

[1804] Example: Based on a specific product promotion scenario, generate video and audio of an actor introducing a product.

[1805] Utilizing the Emotion Engine

[1806] The server uses an emotion engine to analyze user feedback and adjust the generated content. The emotion engine recognizes user emotions in real time and fine-tunes the generative model based on them.

[1807] Example: Collecting the user's emotions as feedback and adjusting the actor's facial expressions based on that data.

[1808] NFT conversion

[1809] The server stores the generated talent's visual and audio information as an NFT, which is recorded on the Ethereum blockchain with a digital signature, making it a unique digital asset.

[1810] Example: The generated video and audio of an actor is registered as an NFT, and its ownership information is recorded on the blockchain.

[1811] Terminal

[1812] Entering and Submitting Data

[1813] The terminal checks the format and content of the visual, audio, and text information entered by the user for errors, and then transmits it to the server.

[1814] Example: A device verifies the format of a movie clip, checks the sampling rate of an interview audio, and ensures a social media post is properly formatted before sending it to a server.

[1815] Receiving and displaying the generated results

[1816] The terminal receives the generated content from the server and displays it to the user, allowing the user to review the results and make correction requests if necessary.

[1817] Example: Display a video of an actor introducing a new product sent from the server and play an audio file.

[1818] Feedback collection with emotion engine

[1819] While the user is viewing the generated content, the device analyzes facial expressions and speech in real time and sends the results to the emotion engine.

[1820] Example: Analyze the facial expressions and speech of the user while watching, and send the emotional data to the emotion engine on the server.

[1821] User

[1822] Talent selection and data provision

[1823] Users log into the system, select a period of time they want to focus on a talent, and upload visual, audio, and text information related to that period.

[1824] Example: A marketer selects an actor's data from the 1990s and uploads film clips, audio interviews, and social media posts from that period.

[1825] Check the generated results and request corrections

[1826] The user can review the generated visual and audio information, request corrections from the server, and even provide feedback using the emotion engine to optimize the generation process.

[1827] Example: The generated video is reviewed, and specific facial expressions or lines are corrected by the server, and the emotion engine learns from the user's feedback.

[1828] final use

[1829] The user ultimately selects the content they are satisfied with and uses it for advertising campaigns and promotions.

[1830] Example: A marketer distributes the final video for an advertising campaign.

[1831] Prompt Sentence Examples

[1832] "Generate a new product promotional video using the face and voice of a famous actor from the 1980s. Use uploaded film clips and interview audio."

[1833] "Please use the user's emotional data as a reference to optimize the facial expressions of the actors in the generated video to make them appear happier."

[1834] The above is an embodiment of the present invention.

[1835] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1836] Step 1: Data collection

[1837] The server receives visual, audio, and text information uploaded by users through their devices. Specifically, it receives HTTP requests and saves image files and text data in storage. The input to this process is the data provided by the user, and the output is the saved raw data. Specifically, the server receives and saves actor film clips, interview audio files, and social media posts sent by marketers.

[1838] Step 2: Data preprocessing (visual information)

[1839] The server preprocesses the received visual information and extracts facial features using a facial recognition algorithm such as FaceNet. The input to this process is the collected visual information, and the output is the extracted facial feature information. Specifically, the server uses OpenCV to analyze frames from a movie clip, detect facial regions, and extract feature points (eyes, nose, mouth, etc.).

[1840] Step 3: Data preprocessing (audio information)

[1841] The server uses FFmpeg and Audacity to preprocess the audio information and remove noise. The input of this process is the collected audio information, and the output is clear audio data with noise removed. Specifically, the server uses FFmpeg to read the audio file, apply a filter to remove noise, and generate clear audio data.

[1842] Step 4: Data preprocessing (text information)

[1843] The server preprocesses the text information and performs tokenization using a natural language processing library (NLTK or spaCy). The input to this process is the collected text information, and the output is tokenized text data. Specifically, the server reads the text data, splits it into tokens, extracts keywords and important phrases, and stores them in a database.

[1844] Step 5: Training the generative model

[1845] The server uses the preprocessed data to train a generative model such as a GAN or Transformer. The input for this process is the preprocessed visual, audio, and text information, and the output is a trained generative model. Specifically, the server uses TensorFlow or PyTorch to train the model using facial features, audio data, and text data as input, and saves the trained model.

[1846] Step 6: Generate AI talent

[1847] The server uses a trained generative model to generate visual and audio information for an AI talent based on a specified scenario. The input to this process is the scenario information and the trained generative model, and the output is the generated visual and audio information. Specifically, the server inputs a prompt sentence (e.g., "Introducing a new product") into the model and saves the generated video and audio.

[1848] Step 7: Leverage your emotional engine

[1849] The device analyzes the user's facial expressions and speech in real time while watching the generated content and sends the results to the emotion engine. The input for this process is the user's facial expression data and voice data, and the output is emotional feedback. Specifically, the device's camera and microphone are used to collect the user's facial expression data and voice data, and the data is sent to the emotion engine in real time.

[1850] The server uses an emotion engine to analyze user feedback and adjusts the generated content by incorporating the results into the generative model. The input to this process is the emotional feedback sent from the device, and the output is the adjusted generated content. Specifically, the emotion engine identifies the user's emotions and optimizes the generated video and audio based on them.

[1851] Step 8: NFT

[1852] The server stores the generated visual and audio information of the talent as an NFT. The input of this process is the generated visual and audio information, and the output is an NFT recorded on the blockchain. Specifically, the server digitally signs the generated data and records it on a blockchain network such as Ethereum.

[1853] (Application example 2)

[1854] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1855] In the prior art, there was no effective method for generating visual and audio information to recreate a specific celebrity's peak performance, and it was difficult to reflect user emotional feedback in real time. Furthermore, there was insufficient protection of the value of the generated content and proof of ownership, making it difficult to continue providing high-quality content that satisfies users. The present invention aims to solve these problems.

[1856] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting visual information, means for collecting audio information, means for collecting text information, means for preprocessing the information, means for training a generative model using the preprocessed information, means for generating visual and audio information using the trained generative model, means for storing the generated information as a non-fungible token, and means for collecting and analyzing user feedback using an emotion engine to allow users to customize the generated visual and audio information and reflecting the feedback in the generation process. This makes it possible to continue providing high-quality, customizable content and also makes it easier to preserve the value of the generated content and prove ownership.

[1857] "Visual information" is a general term for image and video data collected from users and other sources.

[1858] "Audio Information" refers collectively to audio data collected from users and other sources.

[1859] "Text Information" refers collectively to written or typed character information collected from users or other sources.

[1860] "Preprocessing" is the process of formatting collected visual, audio, and text information to make it easier to analyze.

[1861] A "generative model" is a machine learning or deep learning algorithm that uses collected and preprocessed data to generate new visual or audio information.

[1862] A "non-fungible token" is a digital asset with unique value based on a blockchain, typically used for proof of ownership and store of value purposes.

[1863] An "emotion engine" is an algorithm and system that analyzes users' emotions in real time and collects and reflects the results as feedback.

[1864] "User feedback" refers to the user's impressions and reactions to the generated visual and audio information, and requests for modifications based on the impressions and reactions.

[1865] This invention provides a system that collects visual, audio, and text information, preprocesses them, and then generates visual and audio information using a generative model. The system includes a means for storing the generated information as a non-fungible token (NFT). The system also uses an emotion engine to allow users to customize the generated visual and audio information, collects and analyzes user feedback, and incorporates that feedback into the generation process.

[1866] System configuration and operation overview

[1867] 1. Visual information collection and preprocessing:

[1868] The server receives image and video files uploaded by users, such as old movie clips or interview footage.

[1869] Use OpenCV to resize and reshape the image and extract facial features.

[1870] 2. Acquisition and preprocessing of audio information:

[1871] The server receives audio files uploaded by users, for example, interview audio data.

[1872] Use pydub to convert the audio data to mono and format it in a way that is suitable for the speech recognition model.

[1873] 3. Text information collection and preprocessing:

[1874] The server receives text information uploaded by the user, such as a social media post or a script.

[1875] Text information is tokenized and converted into a format that can be input into natural language processing models such as GPT-2.

[1876] 4. Training the generative model and generating visual and audio information:

[1877] The server trains a generative model (e.g., GAN, Transformer) using the preprocessed data.

[1878] A trained generative model is used to generate visual and audio information.

[1879] 5. Feedback collection and analysis using emotion engine:

[1880] The terminal presents the generated visual and audio information to the user and collects the user's emotional feedback in real time.

[1881] The emotion engine analyzes the user's facial expressions and voice and sends the feedback to the server.

[1882] 6. Optimization of generated information and NFTization:

[1883] The server optimizes the generation process based on user feedback.

[1884] The final content generated is stored as a non-fungible token (NFT) to preserve value and prove ownership.

[1885] Specific examples

[1886] If an advertising agency wants to use a famous celebrity from the 1990s to promote a new product, they can use this system. They can upload old movie clips, interview audio, social media posts, etc. to train a generative model. The system then generates an advertising video, and users can provide real-time feedback via smartphone or head-mounted display. The final optimized advertising video is stored as an NFT, preserving ownership and value.

[1887] Example prompt sentence:

[1888] "Generate an advertising video for new product X featuring famous 90s actor A. Use the following information: movie clips, interview audio, and social media posts."

[1889] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1890] Step 1:

[1891] The server collects visual information from users. Specifically, users upload image or video files, such as old movie clips or interview footage. The server receives these files, resizes and reshapes the images using OpenCV, and extracts facial features.

[1892] Input: User-uploaded film clips and interview footage

[1893] Processing: Resize, shape, and extract facial features using OpenCV

[1894] Output: Preprocessed visual data

[1895] Step 2:

[1896] The server collects audio information from users, specifically, by uploading interview recordings or other audio files. The server receives these audio files and uses pydub to convert them to mono and remove background noise.

[1897] Input: Audio file uploaded by the user

[1898] Processing: Mono conversion and noise removal using pydub

[1899] Output: Preprocessed audio data

[1900] Step 3:

[1901] The server collects text information from users. Specifically, users upload text data such as social media posts and scripts. The server receives this text information, tokenizes it, and converts it into a format suitable for input into a natural language processing model (such as GPT-2).

[1902] Input: Text data uploaded by the user

[1903] Processing content: Tokenization, format conversion

[1904] Output: Preprocessed text data

[1905] Step 4:

[1906] The server uses the preprocessed visual, audio, and text data to train a generative model, which is trained using machine learning algorithms such as GAN and Transformer to generate visual and audio information to recreate the talent in their prime.

[1907] Input: Preprocessed visual data, audio data, and text data

[1908] Processing content: Training of generative models (GAN and Transformer)

[1909] Output: Generated visual and audio information

[1910] Step 5:

[1911] The server transmits the generated visual and audio information to the user's device, which then presents the information to the user. The user can then view the generated visual and audio information using a smartphone or head-mounted display and provide emotional feedback.

[1912] Input: Generated visual and audio information

[1913] Processing: Presenting information and collecting emotional feedback

[1914] Output: User's emotional feedback

[1915] Step 6:

[1916] The emotion engine analyzes user feedback, including facial expressions and tone of voice, and optimizes the generated visual and audio information based on the results. Feedback includes the user's facial expressions and tone of voice, and by analyzing these, it adjusts the output of the generative model.

[1917] Input: User's emotional feedback

[1918] Processing content: Analysis of emotional feedback and optimization of generated information

[1919] Output: Optimized visual and audio information

[1920] Step 7:

[1921] The server stores the optimized visual and audio information as non-fungible tokens (NFTs), a process that involves creating digital signatures and recording them on the blockchain.

[1922] Input: Optimized visual and audio information

[1923] Processing: Save as NFT, create digital signature, record on blockchain

[1924] Output: Visual and audio information stored as NFTs

[1925] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1926] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1927] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1928] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1929] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1930] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1931] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1932] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1933] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1934] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1935] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1936] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1937] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1938] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1939] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1940] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1941] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1942] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1943] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1944] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1945] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1946] The following is further disclosed regarding the above embodiment.

[1947] (Claim 1)

[1948] a means for collecting visual information;

[1949] a means for collecting audio information;

[1950] a means for collecting text information;

[1951] a means for preprocessing this information;

[1952] means for training a generative model using the preprocessed information;

[1953] means for generating visual and audio information using a trained generative model;

[1954] A means of storing the generated information as an NFT;

[1955] A system including:

[1956] (Claim 2)

[1957] 10. The system of claim 1, further comprising means for providing the generated visual and audio information in a downloadable form to a user.

[1958] (Claim 3)

[1959] 10. The system of claim 1, further comprising means for error checking the collected visual, audio and text information and transmitting the information to a server.

[1960] "Example 1"

[1961] (Claim 1)

[1962] a means for collecting visual information;

[1963] a means for collecting audio information;

[1964] a means for collecting text information;

[1965] a means for preprocessing this information;

[1966] means for training a generative AI model using the preprocessed information;

[1967] means for generating visual and audio information using a trained generative AI model;

[1968] A means for digitally signing and recording the generated visual and audio information as non-fungible tokens on a blockchain;

[1969] A system including:

[1970] (Claim 2)

[1971] 10. The system of claim 1, further comprising means for providing the generated visual and audio information in a downloadable form to a user, and for allowing the user to review the generated results and request corrections.

[1972] (Claim 3)

[1973] 10. The system of claim 1, further comprising means for error checking the collected visual, audio and text information and transmitting the information to a server.

[1974] "Application Example 1"

[1975] (Claim 1)

[1976] a means for collecting visual information;

[1977] a means for collecting audio information;

[1978] a means for collecting text information;

[1979] a means for preprocessing this information;

[1980] means for training a generative model using the preprocessed information;

[1981] means for gen...

Claims

1. a means for collecting visual information; a means for collecting audio information; a means for collecting text information; a means for preprocessing this information; means for training a generative model using the preprocessed information; means for generating visual and audio information using a trained generative model; A means of storing the generated information as an NFT; A system including:

2. 10. The system of claim 1, further comprising means for providing the generated visual and audio information in a downloadable form to a user.

3. 10. The system of claim 1, further comprising means for error checking the collected visual, audio and text information and transmitting the information to a server.

Images