System

Abstract

To provide a system.SOLUTION: A system includes: means for learning words and actions, action guidelines, a life philosophy, and past books of a specific person; means for generating a video and voice in which the learned specific person is talking with respect to an input question or consultation; means for providing the generated video and voice to inside or outside; means for preprocessing learned data; means for causing a deep learning model to learn using the preprocessed data; means for evaluating the performance of the learned model; and means for generating a response to the question or consultation from a user. The means for generating the response generates a response using a generative AI model on the basis of a prompt sentence in which a feeling state of the user and thinking of the specific person are combined.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] When you want to utilize the knowledge and experience of a specific person, you have the problem that you cannot directly access that person. Also, there are limited means to widely share that person's knowledge and experience. [Means for solving the problem]

[0005] The system includes a means for learning the words and actions, guidelines for action, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person is speaking in response to input questions and inquiries, and a means for providing the generated video / audio internally or externally. This makes it possible to directly utilize the knowledge and experience of a specific person, and also to share that knowledge and experience widely. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16]FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. [Figure 19] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system according to the embodiment 2 of the second embodiment in the case where an emotion engine is combined. [Figure 20] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when combined with an emotion engine. [Figure 21] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the embodiment 3 of the form example 3 in the case where an emotion engine is combined. [Figure 22] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] As one embodiment of the present invention, there is a system equipped with a means for automatically collecting public information of a specific person, such as articles, books, videos, audio, etc. on the Internet. This system learns the words and actions, guidelines for action, and life philosophy of the specific person based on the collected information. Machine learning techniques such as deep learning are used for the learning.

[0029] "Example 2"

[0030] Furthermore, as one embodiment of the present invention, there is a system equipped with a means for generating video / audio that sounds as if a specific person who has been trained is speaking in response to an input question / consultation, etc. This system combines natural language processing technology and audio / video generation technology to generate video / audio that sounds as if a specific person is actually answering the question / consultation.

[0031] "Example 3"

[0032] As one embodiment of the present invention, there is a system equipped with a means for providing the generated video / audio internally or externally. This system makes it possible to utilize the knowledge and experience of a specific person by providing the generated video / audio to members of the company. In addition, by publishing the generated video / audio on a website, it becomes possible to share that knowledge and experience widely.

[0033] The process flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: Automatically collect public information about a specific person, such as articles, books, videos, audio, etc. on the Internet. This collection is done using web scraping technology and APIs.

[0036] Step 2: Based on the collected information, the behavior, code of conduct, and life philosophy of a specific person are learned. This learning is done using machine learning techniques such as deep learning. Specifically, a neural network is trained to generate a model that mimics the behavioral patterns of a specific person.

[0037] "Example 2"

[0038] Step 1: Receive input from the user, such as a question or inquiry. This input can be in text or voice format.

[0039] Step 2: In response to the input question or inquiry, video / audio is generated that sounds as if a specific person who has been trained is speaking. This generation is performed by combining natural language processing technology and audio / video generation technology. Specifically, using the trained model, video / audio is generated that sounds as if a specific person is actually answering the question or inquiry.

[0040] "Example 3"

[0041] Step 1: The generated video / audio is provided internally or externally. This provision can take various forms, such as sharing information with company members or publishing on a website. Specifically, for company members, it is provided via email or the company network, and for publishing on a website, it is provided as a streaming service or download.

[0042] Example 1

[0043] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0044] In conventional systems, data collection and preprocessing to learn a specific person's words and actions, guidelines for action, and life philosophy were often done manually, which was problematic in terms of efficiency. In addition, the performance evaluation of the trained model and the generation of responses to user questions were not sufficiently automated, resulting in a lack of accuracy and reliability. Furthermore, there were limited means to effectively provide the generated video and audio, which made it difficult to improve the user experience.

[0045] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0046] In this invention, the server includes a means for automatically collecting public information related to a specific person from the Internet, a means for preprocessing the collected data, a means for training a deep learning model using the preprocessed data, a means for evaluating the performance of the trained model, and a means for generating responses to questions from users. This makes it possible to automate a series of processes from data collection to model training, performance evaluation, and response generation, and provide an efficient and highly accurate system.

[0047] A "specific person" refers to a specific individual, and refers to information regarding their words and actions, code of conduct, life philosophy, past works, etc.

[0048] "Behavior" refers to words and actions made by a particular person, and reflects that person's characteristics and personality.

[0049] A "code of conduct" refers to the basic policies or principles by which a particular person acts, and indicates that person's values ​​and beliefs.

[0050] A "philosophy of life" refers to a person's thoughts and beliefs about life, and reflects that person's way of life and values.

[0051] "Past books" refer to books written by a particular person or about that person, which record that person's thoughts and activities.

[0052] "Public information" refers to information available on the Internet or from other public sources, including data relating to a particular person.

[0053] "Automated collection means" refers to the use of programs or algorithms to collect publicly available information relating to a particular person without manual intervention.

[0054] "Preprocessing means" refers to processes for converting collected data into a format that is easier to analyze, including data cleaning and normalization.

[0055] A "deep learning model" refers to a machine learning model that uses neural networks and has the ability to learn complex patterns and features.

[0056] "Means of evaluating performance" refers to methods for measuring the accuracy and reliability of a trained model, using evaluation metrics and test data.

[0057] The "means for generating a response" refers to a method for generating an appropriate answer to a question from a user, and creates a response using a learned model.

[0058] "Means for generating video / audio" refers to technology for creating video and audio that sounds like a specific person is speaking, using a generative AI model.

[0059] "Means of providing" refers to the method for providing the generated video and audio to the user, which is carried out through an internal system or an external platform.

[0060] "Means for publishing on a website" refers to a method for publishing the generated video and audio on a website on the Internet, providing them in a format that can be accessed by users.

[0061] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0062] This invention relates to a system that learns the words and actions, guidelines, life philosophies, past books, etc. of a specific person. This system automatically collects public information related to a specific person, preprocesses the collected data, and trains it using a deep learning model. Furthermore, the system evaluates the performance of the trained model and generates responses to questions from users.

[0063] Data collection

[0064] The server automatically collects public information related to a specific person from the Internet. Specifically, it uses Python's BeautifulSoup and Scrapy to scrape web pages and extract the necessary information. For example, it collects data such as articles, books, videos, and audio related to a specific person.

[0065] Data Preprocessing

[0066] The server preprocesses the collected data. For text data, it uses NLTK or spaCy to clean, tokenize, and normalize the text. For audio data, it uses the Google® Cloud Speech-to-Text API to convert the audio data to text. This puts the data in a format that is easy to parse.

[0067] Model Training

[0068] The server trains a deep learning model using the preprocessed data. Using TENSORFLOW (registered trademark) and PyTorch, it builds a model that learns a specific person's words and actions, guidelines for action, and life philosophy. Specifically, it trains models such as LSTM and Transformer using the collected text data as input.

[0069] Model evaluation

[0070] The server evaluates the performance of the trained model using metrics such as precision, recall, and F1 score. Specifically, it evaluates the model's predictive performance using test data and adjusts the model's hyperparameters as necessary.

[0071] Generate a response to the user

[0072] The user inputs a question about a specific person into the generative AI model from the terminal. The server receives the prompt sentence from the user and generates a response using the trained model. For example, if the user inputs "Tell me about the leadership style of a specific person," the server uses the model to generate information about the person's leadership style and provides it to the user.

[0073] Examples of concrete examples and prompts

[0074] For example, if a specific person is a "famous leader," the server will collect articles, books, videos, and audio about that leader from the Internet. The collected data will be preprocessed and the deep learning model will be used to learn the leader's words, actions, and life philosophy.

[0075] Examples of prompts to be input to a generative AI model include:

[0076] "Tell me about the leadership styles of some famous leaders."

[0077] "Describe how a notable leader led themselves to success."

[0078] "Tell me more about the life philosophies of famous leaders."

[0079] In this way, a user can obtain detailed information about a particular person.

[0080] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0081] Program processing flow

[0082] Step 1: Data collection

[0083] The server automatically collects public information related to a specific person from the Internet. Specifically, it uses Python's BeautifulSoup and Scrapy to scrape web pages and extract the necessary information. As input, it receives a list of URLs related to a specific person, and as output, it generates the collected text data and metadata.

[0084] Step 2: Data Preprocessing

[0085] The server preprocesses the collected data. For text data, it uses NLTK or spaCy to clean, tokenize, and normalize the text. For audio data, it uses Google Cloud Speech-to-Text API to convert the audio data to text. As input, it takes the collected raw data and produces preprocessed, clean text data as output.

[0086] Step 3: Model training

[0087] The server trains a deep learning model using the preprocessed data. It uses TensorFlow and PyTorch to build a model that learns a specific person's words and actions, behavioral guidelines, and life philosophy. Specifically, it trains models such as LSTM and Transformer. It receives preprocessed text data as input and generates a trained model as output.

[0088] Step 4: Model evaluation

[0089] The server evaluates the performance of the trained model. For the evaluation, it uses metrics such as precision, recall, and F1 score. Specifically, it evaluates the predictive performance of the model using test data and adjusts the model's hyperparameters as necessary. It receives the trained model and test data as input and generates the evaluation results as output.

[0090] Step 5: Accepting user questions

[0091] A user inputs a question about a specific person into the generative AI model from a terminal. The model receives the user's question as input and sends the question to the server as output.

[0092] Step 6: Response Generation

[0093] The server receives a prompt from the user and generates a response using the trained model. Specifically, the server inputs the user's question into the model and generates an appropriate answer. The server receives the user's question and the trained model as input, and generates the generated response as output.

[0094] Step 7: Providing a response

[0095] The server provides the generated response sentence to the user. Specifically, the server transmits the generated response sentence to the terminal so that the user can view it. The server receives the generated response sentence as input and displays the response sentence on the user's terminal as output.

[0096] In this way, the entire process from data collection to response generation is automated, resulting in an efficient and highly accurate system.

[0097] (Application example 1)

[0098] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] In modern society, there is a demand to learn the words and actions, guidelines for action, and life philosophies of specific people and apply them to daily life and work. However, there is a lack of efficient means to collect this information and provide it to users in an appropriate format. Furthermore, there is no system that automatically collects public information about specific people and uses it to provide specific advice and motivation to users. As a result, users must spend a great deal of time and effort collecting and analyzing information.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0101] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question or inquiry, a means for providing the generated video / audio internally or externally, a means for automatically collecting public information about the specific person, and a means for providing the user with advice and motivation that is useful in daily life and work based on the collected information. This allows the user to efficiently learn the philosophy and guidelines of a specific person and use them in daily life and work.

[0102] A "specific person" refers to a specific individual, and refers to information about that person's words and actions, code of conduct, life philosophy, past books, etc.

[0103] "Words and actions" refers to what a particular person says or does.

[0104] A "code of conduct" refers to the basic ideas or policies that a particular person follows when acting.

[0105] "Philosophy of life" refers to a person's basic thoughts and values ​​regarding life.

[0106] "Past books" refers to books written by a particular person or books about that person.

[0107] "Means of learning" refers to a method of learning using machine learning technology based on information such as a specific person's words and actions, code of conduct, philosophy of life, and past books.

[0108] "Question / Inquiry" refers to a question or inquiry entered by a user for a specific person.

[0109] "Means for generating video / audio" refers to means for generating video and audio that sounds like a specific trained person is speaking.

[0110] The "means for providing" refers to the means for providing the generated video and audio internally or externally.

[0111] "Means of automatically collecting public information" refers to means of automatically collecting public information such as articles, books, videos, and audio on the Internet.

[0112] "Means of providing advice and motivation" refers to means of providing users with advice and motivation that is useful in their daily lives and work, based on the collected information.

[0113] A system for implementing this invention includes means for learning the words and actions, guidelines for action, life philosophy, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person is speaking in response to an input question or inquiry, means for providing the generated video / audio internally or externally, means for automatically collecting public information about the specific person, and means for providing the user with advice and motivation that is useful in their daily life and work based on the collected information.

[0114] System Configuration

[0115] Hardware

[0116] Server: A central processing unit for collecting, learning, generating, and providing data.

[0117] Smartphone: A device for users to input questions and inquiries and receive the generated video / audio

[0118] software

[0119] Python: A programming language for data collection, machine learning, and generative processing

[0120] BeautifulSoup: A library for scraping articles and information from the internet

[0121] TensorFlow: A library for building and training deep learning models

[0122] Data processing and calculation

[0123] Data collection

[0124] The server automatically collects public information about a particular person by using BeautifulSoup to scrape information such as articles, books, videos, and audio from the Internet.

[0125] Data Preparation

[0126] Based on the collected information, we create a dataset to learn the words and actions, behavioral guidelines, and life philosophies of a particular person, which includes preprocessing and labeling the text data.

[0127] Model Building and Training

[0128] Using TensorFlow, a Long Short-Term Memory (LSTM) model is built and trained on the collected data, creating a generative AI model that mimics the behavior and course of action of a specific person.

[0129] Video / audio generation

[0130] Using the learned model, video / audio that sounds like a specific person is speaking in response to a question or inquiry entered by the user is generated.

[0131] offer

[0132] The generated video and audio are sent from the server to the user's smartphone, allowing the user to efficiently learn the philosophy and code of conduct of a particular person and apply it to their daily life and work.

[0133] Examples

[0134] When a user inputs the name of a "certain celebrity" into their smartphone, the server collects publicly available information about that person and learns from it. Then, when the user inputs a question such as "What would a particular celebrity think in this situation?", the server uses the learned model to generate video and audio that sounds like the person is speaking and sends it to the smartphone.

[0135] Examples of prompt statements

[0136] Enter the name of a specific celebrity. For example, "Specific Celebrity A" or "Specific Celebrity B".

[0137] In this way, users can learn the philosophies and codes of conduct of a particular person and apply them in their daily lives and work.

[0138] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0139] Step 1:

[0140] A user inputs the name of a specific person into a smartphone terminal. The input name is sent to a server. The input data is the name of a specific person.

[0141] Step 2:

[0142] The server automatically collects public information such as articles, books, videos, and audio on the Internet based on the input name of a specific person. Specifically, it uses BeautifulSoup to perform web scraping to obtain related text data. The input data is the name of a specific person, and the output data is the collected public information.

[0143] Step 3:

[0144] The server creates a dataset based on the collected public information to learn the words and actions, behavioral guidelines, and life philosophies of a specific person. It preprocesses the text data (tokenization, normalization, labeling, etc.) and formats the learning data. The input data is the collected public information, and the output data is the preprocessed learning data.

[0145] Step 4:

[0146] The server uses TensorFlow to build a Long Short-Term Memory (LSTM) model and trains the model based on preprocessed training data. This creates a generative AI model that mimics the words, actions, and behavior of a specific person. The input data is the preprocessed training data, and the output data is the trained generative AI model.

[0147] Step 5:

[0148] A user inputs a question or inquiry into a smartphone terminal. The input question or inquiry is sent to a server. The input data is the content of the user's question or inquiry.

[0149] Step 6:

[0150] The server uses a trained generative AI model to generate video / audio that sounds like a specific person is speaking in response to a question or inquiry input by a user. The input data is the user's question or inquiry content and the trained generative AI model, and the output data is the generated video / audio.

[0151] Step 7:

[0152] The server transmits the generated video / audio to the smartphone terminal. The user can receive and view the generated video / audio on the smartphone terminal. The input data is the generated video / audio, and the output data is the video / audio displayed on the user's smartphone terminal.

[0153] In this way, users can learn the philosophies and code of conduct of a particular person and apply them to their daily lives and work.

[0154] Example 2

[0155] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0156] In conventional systems, it was difficult to generate video and audio that sounded like a specific person was actually answering a user's question or inquiry. In addition, the quality of the generated video and audio was low, making it impossible to provide a realistic experience to the user. Furthermore, there was a lack of means to learn the words, actions, and guidelines of a specific person, so the generated answers lacked reliability and consistency.

[0157] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0158] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for analyzing input text using natural language processing technology for input questions and inquiries, a means for generating a prompt sentence based on the analysis result, a means for inputting the prompt sentence into a generation AI model and generating an answer text, a means for generating audio in the voice of a specific person using speech synthesis technology from the generated answer text, a means for generating an image of a specific person speaking using video generation technology based on the generated audio, and a means for providing the generated video / audio internally or externally. This makes it possible to generate high-quality video and audio as if a specific person were actually answering questions and inquiries from users, providing a realistic experience.

[0159] A "specific person" is a specific individual designated by the user, and is a person whose words and actions, code of conduct, philosophy of life, past books, etc. are to be studied.

[0160] "Behavior" refers to the words and actions taken by a particular person and reflects that person's characteristics and personality.

[0161] A "code of conduct" is a standard or policy of behavior that a particular person should follow, and indicates that person's values ​​and beliefs.

[0162] A "philosophy of life" is a person's thoughts and beliefs about life, which reflect that person's way of life and values.

[0163] "Past books" refer to books written by a particular person or books related to that person, and are materials for learning about that person's ideas and knowledge.

[0164] "Natural language processing technology" refers to technology that enables computers to understand and analyze human language, analyzing text and extracting meaning.

[0165] A "prompt" is a document to be input into a generative AI model, and is an instruction text for generating answers to users' questions or inquiries.

[0166] A "generative AI model" is a model that uses artificial intelligence to generate text or speech, and generates answers based on prompts.

[0167] "Speech synthesis technology" refers to technology that converts text into speech and is used to generate speech in the voice of a specific person.

[0168] "Image generation technology" refers to technology that generates images based on audio or text, and is used to create images of a specific person speaking.

[0169] "Means for providing internally or externally" refers to means for storing the generated video and audio within the system, or for providing them to an external user or system.

[0170] The present invention is a system for generating video and audio that sounds as if a specific person is actually answering a question or inquiry from a user. This system is implemented using the following hardware and software.

[0171] Hardware and Software Used

[0172] Hardware: Server equipped with a high-performance GPU

[0173] Software: Natural Language Processing (NLP) libraries (e.g. spaCy, Transformers), voice synthesis software (e.g. Tacotron 2), video generation software (e.g. DeepFake technology)

[0174] Specific operation of the system

[0175] When the server receives a question or request for advice from a user, it first analyzes the input text using natural language processing technology. The analysis includes grammatical analysis and keyword extraction. It then generates a prompt sentence based on the analysis results. For example, if a user enters, "I haven't been able to sleep at night lately. What should I do?", the following prompt sentence will be generated:

[0176] Example prompt:

[0177] "Health Advice: I haven't been able to sleep at night lately, what should I do? Let your doctor answer this question."

[0178] The server inputs this prompt sentence into a generative AI model (e.g., GPT-3 (registered trademark)) to generate an answer text. The generated answer text may look something like the following:

[0179] Example of a generated answer:

[0180] "There are several possible reasons why you can't sleep at night. First, it's important to make time to relax before going to bed. For example, taking a warm bath or doing some light stretching is effective. It's also important to limit your caffeine intake."

[0181] Next, the server inputs the generated response text into speech synthesis software to generate audio in a specific person's voice. For example, audio is generated in the voice of a doctor. Then, based on the generated audio, video generation software is used to generate video of the specific person speaking. Finally, the generated audio and video are integrated into a single video file.

[0182] Examples

[0183] For example, if a user enters "I haven't been able to sleep at night lately. What should I do?", the process will proceed as follows:

[0184] 1. User input: The user uses a terminal to enter a question.

[0185] 2. Text analysis: The server uses an NLP library to extract keywords such as "I can't sleep at night" and "What should I do?"

[0186] 3. Prompt generation: The server generates a prompt such as "Health consultation: I haven't been able to sleep at night lately. What should I do? Please have a doctor answer this question."

[0187] 4. Input to the generative AI model: The server inputs the prompt sentence into GPT-3.

[0188] 5. Answer generation: GPT-3 generates the answer, "There are several possible reasons why you can't sleep at night. First, it is important to take time to relax before going to bed. For example, taking a warm bath or doing some light stretching can be effective. It is also important to limit your caffeine intake."

[0189] 6. Speech synthesis: The server uses Tacotron 2 to generate speech in the doctor’s voice.

[0190] 7. Video Generation: The server uses DeepFake technology to generate a video of the doctor speaking.

[0191] 8. Audio and video integration: The server integrates the generated audio and video.

[0192] 9. Result delivery: The server sends the consolidated video file to the user's device, where the user watches the video.

[0193] In this way, the user can experience high quality video and audio as if a specific person were actually answering the question.

[0194] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0195] Step 1:

[0196] Users access the system using a terminal and input their questions or inquiries. The input text is then sent to the server.

[0197] Input: User's question or concern (e.g. "I haven't been able to sleep at night lately. What should I do?")

[0198] Output: The input text sent to the server.

[0199] Step 2:

[0200] The server analyzes the received input text using a natural language processing (NLP) library (e.g., spaCy), which includes grammar analysis and keyword extraction.

[0201] Input: The input text sent by the user

[0202] Data processing: grammatical analysis, keyword extraction

[0203] Output: Analysis results (e.g. keywords such as "I can't sleep at night" and "What should I do")

[0204] Step 3:

[0205] The server generates a prompt sentence based on the analysis results. The prompt sentence is an instruction sentence to be input to the generative AI model.

[0206] Input: Analysis results

[0207] Data processing: Prompt generation

[0208] Output: Prompt text (e.g. "Health question: I've been having trouble sleeping at night lately. What should I do? Please have your doctor answer this question.")

[0209] Step 4:

[0210] The server inputs the generated prompt sentence into a generative AI model (e.g., GPT-3) to generate an answer text.

[0211] Input: prompt statement

[0212] Data computation: Generative AI model generates answers

[0213] Output: Answer text (Example: "There are several possible reasons why you can't sleep at night. First, it's important to take time to relax before going to bed. For example, taking a warm bath or doing some light stretching can be effective. It's also important to limit your caffeine intake.")

[0214] Step 5:

[0215] The server inputs the generated answer text into speech synthesis software (e.g., Tacotron 2) to generate a voice in the voice of a specific person.

[0216] Input: Answer text

[0217] Data Calculation: Voice Synthesis

[0218] Output: Audio data (e.g. generated audio of a doctor's voice)

[0219] Step 6:

[0220] The server uses video generation software (e.g., DeepFake technology) based on the generated audio to generate a video of a specific person speaking.

[0221] Input: Audio data

[0222] Data Calculation: Image Generation

[0223] Output: Video data (e.g. video of a doctor speaking)

[0224] Step 7:

[0225] The server combines the generated audio and video into a single video file.

[0226] Input: Audio data, video data

[0227] Data processing: Audio and video integration

[0228] Output: A merged video file

[0229] Step 8:

[0230] The server transmits the combined video file to the user's terminal, and the user watches the video.

[0231] Input: merged video file

[0232] Output: Video file sent to user's device

[0233] The above is the specific processing flow of the program of this system.

[0234] (Application example 2)

[0235] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0236] In conventional customer service in brick-and-mortar stores, when a customer has a question or inquiry, a sales clerk is required to respond, and because this depends on the number and skills of the sales clerks, the response may be delayed or an insufficient answer may be provided. Furthermore, if a specific sales clerk is not available or there are only a limited number of sales clerks with specialized knowledge, customer satisfaction may decrease. To solve these problems, there is a demand for a system that can generate video and audio that sounds like a specific person is speaking, and have a virtual sales clerk serve customers in a brick-and-mortar store.

[0237] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for learning the words and actions, guidelines for action, philosophy of life, past books, etc. of a specific person, a means for generating video / audio that sounds like the learned specific person is speaking in response to an input question / inquiry, etc., a means for providing the generated video / audio internally or externally, a means for generating video / audio of a virtual store clerk answering a customer's question in a real store, and a means for providing the generated video / audio to a smartphone. This makes it possible for a customer in a real store to get a quick and accurate answer from a virtual store clerk using a smartphone.

[0238] A "specific person" refers to an individual whose words and actions, code of conduct, life philosophy, past books, etc. are the subject of study.

[0239] "Speech and behavior" refers to the words and actions of a particular person.

[0240] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[0241] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[0242] "Past books" refers to books and publications written by a specific person.

[0243] "Means of learning" refers to methods and techniques for collecting and analyzing a specific person's words and actions, code of conduct, philosophy of life, past books, etc. as data.

[0244] "Question / Inquiry" refers to any inquiry or request for advice that a User enters into the System.

[0245] "Means for generating video / audio" refers to technology or methods for producing video and audio that sounds as if a specific trained person is actually speaking.

[0246] "Means of providing" refers to the methods and technologies for delivering the generated video and audio to users internally or externally.

[0247] "Brick and mortar store" refers to a commercial establishment that exists in a physical location.

[0248] A "virtual store clerk" refers to a virtual store clerk created to serve customers in a physical store.

[0249] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and run applications.

[0250] A system for implementing this invention includes means for learning the words and actions, code of conduct, philosophy of life, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person learned is speaking in response to input questions, inquiries, etc., means for providing the generated video / audio internally or externally, means for generating video / audio of a virtual store clerk answering customer questions in a physical store, and means for providing the generated video / audio to a smartphone.

[0251] Hardware and Software Used

[0252] Hardware: Smartphones, servers

[0253] Software: Python, OpenAI(R) API, OpenCV, pyttsx3

[0254] Data processing and calculation

[0255] 1. The server collects data such as a specific person's words and actions, guidelines, life philosophy, past books, etc., and trains the system using natural language processing technology.

[0256] 2. The user uses their smartphone to enter their question or inquiry.

[0257] 3. The server uses the OpenAI API to generate answers to the questions entered, as if the specific person were actually answering them.

[0258] 4. The server converts the generated answer into an audio file using pyttsx3.

[0259] 5. The server uses OpenCV to combine the audio file with the pre-recorded video of the store clerk to generate the final video.

[0260] 6. The server provides the generated video / audio to the smartphone.

[0261] Examples

[0262] For example, when a user enters a question such as "What are the features of this product?" at a physical store, the server performs the following processing.

[0263] 1. The server uses the OpenAI API to generate an answer to the question, "What are the features of this product?"

[0264] 2. The generated answer will be, "This product uses the latest technology and is very high performance."

[0265] 3. The server converts this response into an audio file using pyttsx3.

[0266] 4. The server uses OpenCV to combine this audio file with the pre-recorded video of the store clerk to generate the final video.

[0267] 5. Users can view and hear the virtual store clerk on their smartphones.

[0268] Examples of prompt statements

[0269] Q: What are the features of this product?

[0270] A: This product uses the latest technology and is very powerful.

[0271] In this way, customers can use their smartphones to get quick and accurate answers from virtual store clerks in physical stores.

[0272] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0273] Step 1:

[0274] The user inputs a question or inquiry using a smartphone. The input question is sent to the server as text data.

[0275] Step 2:

[0276] The server sends the received question to the OpenAI API and uses a generative AI model to generate an answer to the question. The input is the user's question text and the output is the generated answer text.

[0277] Step 3:

[0278] The server converts the generated response text into an audio file using pyttsx3. The input is the response text, and the output is the audio file (e.g., response.mp3).

[0279] Step 4:

[0280] The server uses OpenCV to combine the pre-recorded video of the store attendant with the audio file to generate the final video. The input is the pre-recorded video file and the generated audio file, and the output is the final video file (e.g., response_video.mp4).

[0281] Step 5:

[0282] The server provides the generated video / audio to a smartphone. The input is the final video file, and the output is the video / audio played on the smartphone.

[0283] Step 6:

[0284] The user sees the video and audio of the virtual store assistant on his / her smartphone. The input is the video / audio file provided by the server, and the output is the user's visual and auditory reception of the information.

[0285] Example 3

[0286] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0287] In conventional systems, the means to generate the knowledge and experience of a specific person as video and audio and share it both inside and outside the company were limited. It was also difficult to efficiently store the generated video and audio and provide it at the appropriate time. This made it impossible to make the most of the knowledge and experience of a specific person.

[0288] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0289] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question / consultation, etc., a means for inputting a prompt sentence into the generative AI model, a means for the generative AI model to generate video / audio, a means for saving the generated video / audio, a means for providing the generated video / audio internally or externally, a means for providing the generated video / audio to company members, and a means for publishing the generated video / audio on a website. This makes it possible to efficiently generate the knowledge and experience of a specific person as video and audio and share it both inside and outside the company.

[0290] A "specific person" refers to an individual with specific knowledge or experience.

[0291] "Speech and behavior" refers to the words and actions of a particular person.

[0292] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[0293] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[0294] "Past books" refers to books and documents written by a specific person.

[0295] "Means of learning" refers to a method for incorporating a particular person's words and actions, guidelines for behavior, life philosophy, past books, etc. into the system.

[0296] "Questions and inquiries" refers to inquiries or requests for advice that users input into the system.

[0297] "Means for generating video / audio" refers to a method for creating video and audio that sounds like a specific person is speaking based on an input question or inquiry.

[0298] A "generative AI model" refers to a model that uses artificial intelligence to generate images and sounds.

[0299] A "prompt sentence" refers to an input sentence that gives instructions to a generative AI model.

[0300] "Means of storage" refers to the method of storing the generated video and audio as data.

[0301] "Means of providing" refers to the method of distributing the generated video and audio internally or externally.

[0302] "Means of providing to internal members" refers to the method of sharing the generated video and audio with internal stakeholders.

[0303] "Means of publishing on a website" refers to a method of widely publishing the generated video and audio on the Internet.

[0304] This invention relates to a system that generates a specific person's knowledge and experience as video and audio and shares them both inside and outside the company. This system includes a means for learning the specific person's words and actions, guidelines for action, life philosophy, past books, etc., a means for generating video / audio that sounds like the specific person is speaking in response to an input question or consultation, a means for inputting a prompt sentence into a generative AI model, a means for the generative AI model to generate video / audio, a means for saving the generated video / audio, a means for providing the generated video / audio internally or externally, a means for providing the generated video / audio to company members, and a means for publishing the generated video / audio on a website.

[0305] The user inputs a prompt for the generative AI model using a dedicated application or web interface. For example, the user might input, "Please create a promotional video for our new product." The device then sends the input prompt to the server using the HTTPS protocol.

[0306] The server receives the prompt sentence sent from the terminal and stores it in a temporary database. The server then inputs the stored prompt sentence into the generative AI model and starts the generation process. As the generative AI model, for example, OpenAI's GPT-4 (registered trademark) or DALL-E is used. The generative AI model generates video and audio based on the prompt sentence. This generation process can take several seconds to several minutes.

[0307] The generated video and audio are stored in the server's storage. For example, cloud storage such as Amazon S3 or Google Cloud Storage is used as the storage location. The server uploads the saved video and audio to an internal shared folder to provide it to company members, or makes it available on an internal web portal.

[0308] Furthermore, the server publishes the generated video and audio on a website, using a website management system (e.g., WordPress or Joomla) to publish the content.

[0309] As a concrete example, consider the case where a user inputs a prompt saying, "I want you to generate an interview video about a specific project." When this prompt is sent to the server, the server inputs this prompt into the generative AI model. The generative AI model generates an interview video about the project and returns it to the server. The server uploads the generated video to a shared folder within the company so that members within the company can access it. The server can also publish the video on an external website to share it widely.

[0310] Examples of prompts include:

[0311] "I want you to generate interview footage for a specific project."

[0312] "I want you to create a video to introduce our new product."

[0313] "I want you to generate a commentary audio about a specific technology."

[0314] In this way, the user can utilize the generative AI model to generate video and audio according to specific needs and share them inside and outside the company. The flow of the specification process in the third embodiment will be described with reference to FIG.

[0315] Step 1:

[0316] The user inputs a prompt for the generative AI model using a dedicated application or web interface. For example, the user might input, "Please create a promotional video for our new product." The input prompt is temporarily stored in the device's memory.

[0317] Step 2:

[0318] The terminal sends the input prompt text to the server using the HTTPS protocol. The prompt text is encrypted before being sent. The input is the prompt text, and the output is a notification of completion of transmission to the server.

[0319] Step 3:

[0320] The server receives the prompt text sent from the terminal and stores it in a temporary database. The input is the prompt text, and the output is a notification that the text has been saved to the database.

[0321] Step 4:

[0322] The server inputs the saved prompt sentence into the generative AI model and starts the generation process. The input is the prompt sentence, and the output is a notification to the generative AI model that the input has been completed.

[0323] Step 5:

[0324] A generative AI model generates video and audio based on a prompt. This generation process can take anywhere from a few seconds to a few minutes. The input is the prompt, and the output is the generated video and audio.

[0325] Step 6:

[0326] The server receives the video and audio returned from the generative AI model and stores them in the server's storage. The storage location can be cloud storage such as Amazon S3 or Google Cloud Storage. The input is the generated video and audio, and the output is a notification that the storage has been completed.

[0327] Step 7:

[0328] The server uploads the stored video and audio to a shared folder within the company to provide them to company members, or publishes them on a web portal for internal use only. The input is the stored video and audio, and the output is a notification to company members that the video and audio have been provided.

[0329] Step 8:

[0330] The server publishes the generated video and audio on a website. At this time, the content is published using a website management system (e.g., WordPress or Joomla). The input is the saved video and audio, and the output is a notification on the website that publication has been completed.

[0331] (Application example 3)

[0332] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0333] In modern society, there is a demand to widely share the knowledge and experience of specific experts and experienced people. However, there is a lack of means to efficiently collect this knowledge and experience, generate it as video or audio, and provide it to in-house members and general users. This makes it difficult to effectively communicate specialized knowledge and experience.

[0334] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the learned specific person is speaking in response to an input question / consultation, etc., a means for providing the generated video / audio internally or externally, a means for collecting knowledge and experience of specific experts and experienced people in text format, a means for creating a prompt sentence to be input to the generation AI model based on the collected text data, a means for inputting the prompt sentence to the generation AI model and generating video and audio, and a means for publishing the generated video and audio to in-house members or on a website. This makes it possible to efficiently collect the knowledge and experience of specific experts and experienced people, generate it as video and audio, and share it widely.

[0335] A "specific person" refers to a specific expert or person with experience whose knowledge or experience is important.

[0336] "Speech and behavior" refers to the words and actions of a particular person.

[0337] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[0338] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[0339] "Past books" refers to books and documents written in the past by a particular person.

[0340] "Means of learning" refers to methods or techniques for making the system learn a particular person's words and actions, code of conduct, philosophy of life, past books, etc.

[0341] "Input questions and inquiries" refers to questions and inquiries that users make to the system.

[0342] "Means for generating video / audio" refers to methods or technologies for generating video or audio that sounds like a specific trained person is speaking.

[0343] "Means of providing" refers to the methods and technologies for providing the generated video and audio internally or externally.

[0344] "Means of collecting knowledge and experience in text format" refers to methods and technologies for collecting the knowledge and experience of specific experts or experienced people as text data.

[0345] A "generative AI model" refers to an artificial intelligence model that generates video and audio based on an input prompt.

[0346] A "prompt" refers to an instruction or question to be input into a generative AI model.

[0347] "Internal members" refers to members within a particular organization or company.

[0348] "Website" refers to a web page that provides publicly available information on the Internet.

[0349] A system for implementing this invention includes means for learning the words and actions, code of conduct, life philosophy, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person who has learned is speaking in response to input questions, inquiries, etc., means for providing the generated video / audio internally or externally, means for collecting the knowledge and experience of specific experts or experienced people in text form, means for creating prompt text to be input into a generative AI model based on the collected text data, means for inputting the prompt text into the generative AI model and generating video and audio, and means for publishing the generated video and audio to internal members or on a website.

[0350] Hardware and Software Used

[0351] Hardware: Servers, smartphones, head-mounted displays

[0352] Software: Generative AI models (e.g. OpenAI GPT-4), video generation software (e.g. Adobe Premiere Pro), audio generation software (e.g. Audacity)

[0353] Data processing and calculation

[0354] 1. Data collection: The server collects the knowledge and experience of specific experts or experienced people in text format. For example, a doctor's surgical experience is collected as text data.

[0355] 2. Prompt generation: The server creates a prompt sentence to be input to the generative AI model based on the collected text data. For example, it generates a prompt sentence such as "Please explain the procedure of this surgery in detail."

[0356] 3. Video / audio generation: The server inputs the prompt text into the generative AI model to generate video and audio that sounds like a specific expert is speaking.

[0357] 4. Data provision: The server makes the generated video and audio available to company members and on a website, making it possible to share knowledge and experience widely.

[0358] Examples

[0359] For example, a doctor's surgical experience is collected and a prompt sentence that describes the surgical procedure in detail is generated. By inputting the prompt sentence "Please describe the procedure of this surgery in detail. Specifically, please include each step from preparation to completion of the surgery" into the generative AI model, video and audio of the surgical procedure are generated. The generated video and audio are published on a website for medical professionals and shared widely.

[0360] In this way, it is possible to efficiently collect the knowledge and experience of specific experts and experienced people, generate it as video and audio, and share it widely.

[0361] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0362] Step 1:

[0363] The server collects the knowledge and experience of specific experts and experienced people in text format. Specifically, it collects interviews with experts, past literature, lecture contents, etc. as text data. The input is information about the expert's knowledge and experience, and the output is text data.

[0364] Step 2:

[0365] The server creates prompt sentences to be input to the generative AI model based on the collected text data. Specifically, it analyzes the collected text data, extracts important keywords and phrases, and generates prompt sentences. The input is the collected text data, and the output is the generated prompt sentence.

[0366] Step 3:

[0367] The server inputs a prompt sentence into the generative AI model to generate video and audio. Specifically, the server inputs a prompt sentence into the generative AI model and executes a process in which the model generates video and audio. The input is the generated prompt sentence, and the output is the generated video and audio.

[0368] Step 4:

[0369] The server makes the generated video and audio available to company members or on a website. Specifically, it converts the generated video and audio into an appropriate format and uploads it to an internal shared system or website. The input is the generated video and audio, and the output is the published video and audio.

[0370] Step 5:

[0371] A user views publicly available video and audio and learns from the knowledge and experience of a specific expert or person with experience. Specifically, a user accesses a website and plays the publicly available video and audio. The input is the publicly available video and audio, and the output is the user's learning outcome.

[0372] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0373] "Example 1"

[0374] One embodiment of the present invention is a system that incorporates an emotion engine. This system automatically collects public information about a specific person and learns the person's words and actions, guidelines for action, and philosophy of life. Furthermore, the emotion engine that recognizes the user's emotions is used to respond according to the user's emotions. Specifically, when the user is feeling happy, a video / audio is generated in which a specific person shares the joy, and when the user is feeling sad, a video / audio is generated in which a specific person comforts the user.

[0375] "Example 2"

[0376] The generated video / audio can be provided internally or externally. Specifically, for internal members, it can be provided via email or the internal network, and published on a website as a streaming service or available for download. At this time, it is also possible to change the content provided according to the user's emotions. For example, when a user is feeling happy, content is provided in which a specific person shares the joy, and when the user is feeling sad, content is provided in which a specific person comforts the user.

[0377] "Example 3"

[0378] Furthermore, the emotion engine can recognize changes in the user's emotions in real time and generate video / audio in response to those changes. For example, if a user starts out feeling happy but then changes to sadness, the emotion engine will recognize that change and generate video / audio of a specific person comforting them. This allows for an appropriate response to the user's emotions.

[0379] The process flow of each embodiment will be described below.

[0380] "Example 1"

[0381] Step 1: Automatically collect publicly available information about a specific person.

[0382] Step 2: Based on the information you have collected, learn about the person's words and actions, code of conduct, and life philosophy.

[0383] Step 3: Using an emotion engine that recognizes the user's emotions, respond according to the user's emotions.

[0384] Step 4: Generate video / audio that sounds like a specific person is speaking depending on the user's emotions.

[0385] "Example 2"

[0386] Step 1: Provide the generated video / audio internally or externally.

[0387] Step 2: Change what you offer based on the user's emotions.

[0388] Step 3: Depending on the user's emotions, provide content that sounds like a specific person is speaking.

[0389] "Example 3"

[0390] Step 1: Recognize changes in user emotions in real time.

[0391] Step 2: Generate video / audio according to changes in emotions.

[0392] Step 3: Respond appropriately to the user's emotions.

[0393] Example 1

[0394] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0395] Conventional systems have had difficulty not only learning a specific person's words and actions, code of conduct, and life philosophy, but also responding according to the user's emotions. In addition, there was a lack of technology to recognize the user's emotions and generate images and sounds accordingly. This made it difficult to provide a more personalized experience for users.

[0396] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0397] In this invention, the server includes a means for learning the words and actions, guidelines for action, life philosophy, past books, etc. of a specific person, a means for automatically collecting public information of the specific person, a means for recognizing the user's emotions, a means for generating video / audio that sounds like the specific person is speaking according to the user's emotions, and a means for providing the generated video / audio internally or externally. This makes it possible to provide personalized video and audio according to the user's emotions.

[0398] A "specific person" refers to a particular individual or character, whose words and actions, code of conduct, and life philosophy are the subject of study.

[0399] "Behavior" refers to the words and actions of a particular person, and represents that person's characteristics and personality.

[0400] A "code of conduct" refers to the principles or policies of behavior that a particular person follows, which serve as the standard for that person's decision-making and actions.

[0401] "Philosophy of life" refers to a particular person's outlook on life and values, which form the basis of that person's way of life and way of thinking.

[0402] "Past books" refer to books written by a particular person or books about that person, and serve as sources of information for learning.

[0403] "Public information" refers to information that is available to the public, such as Internet articles, books, videos, and audio.

[0404] "Means for recognizing emotions" refers to technology or devices for analyzing a user's voice and facial expression data and identifying the emotion.

[0405] "Means for generating video / audio" refers to technology or equipment for creating video or audio that sounds like a specific person is speaking.

[0406] "Means of providing" refers to the technology and devices for providing the generated video and audio to the user.

[0407] "Internal or external" refers to devices or networks within the system, or the external environment, such as the Internet.

[0408] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0409] The present invention is a system that learns the words and actions of a specific person, guidelines for action, life philosophy, past books, etc., and generates and provides images and sounds according to the user's emotions. A specific embodiment of this system will be described below.

[0410] 1. Program Generation

[0411] The server automatically collects public information about a specific person and generates a program that learns from that information. This program targets articles, books, videos, audio, etc. on the Internet. The software used includes BeautifulSoup and Scrapy as web crawlers, MySQL (registered trademark) and MongoDB as databases, and TensorFlow and PyTorch as deep learning frameworks.

[0412] 2. Program Processing Explanation

[0413] The server uses the generated program to collect public information about a specific person. The collected information is stored in a database. The server then uses deep learning technology to analyze the collected information and learn the specific person's words and actions, code of conduct, and life philosophy.

[0414] In addition, the server runs a system that combines an emotion engine that analyzes the user's voice and facial expression data to recognize the user's emotions. The emotion engine uses software such as OpenCV and Emotion API.

[0415] When a user accesses the system, the terminal transmits the user's emotions to the emotion engine. The emotion engine analyzes the user's emotions and transmits the results to the server. Based on the analysis results, the server generates images and audio for a specific person to respond according to the user's emotions. A generative AI model is used to generate images and audio based on the prompt text.

[0416] 3. Specific Examples

[0417] As a concrete example, consider the case where a specific person is a famous author. The server collects interview articles and lecture videos of the author, and uses deep learning to learn the author's life philosophy. When a user accesses the system and the emotion engine recognizes the user's joy, the server generates video and audio that show the author sharing his or her joy, and sends them to the terminal.

[0418] Example prompt:

[0419] "Collect interviews and lecture videos of famous authors and learn their life philosophies. If the user feels joy, generate video or audio of the author sharing his or her joy."

[0420] In this way, the system operates in cooperation with the server, terminals, and users.

[0421] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0422] Step 1: Gather public information

[0423] The server collects public information about a particular person from the Internet. As input, it uses the name of the particular person and related keywords. Specifically, it uses a web crawler (e.g., BeautifulSoup or Scrapy) to collect information such as articles, books, videos, and audio. As output, it obtains the collected information in the form of text, images, audio, etc.

[0424] Step 2: Save your data

[0425] The server stores the collected information in a database. As input, it uses the information collected in step 1. Specifically, it executes SQL queries to a database such as MySQL or MongoDB to store text data and metadata. As output, it gets the information stored in the database.

[0426] Step 3: Analyze and train data

[0427] The server analyzes the stored data and learns the words and actions, behavioral guidelines, and life philosophy of a particular person. It uses the information stored in the database as input. Specifically, it uses TensorFlow and PyTorch to build a deep learning model and trains the data. The output is a trained model.

[0428] Step 4: Recognize emotions

[0429] The device collects the user's voice and facial expression data and sends it to the emotion engine. As input, the device uses the user's voice and facial expression data. Specifically, the device captures the data using a camera and microphone, and analyzes the emotions using OpenCV and the Emotion API. As output, the analyzed emotion data is obtained.

[0430] Step 5: Generate correspondence

[0431] Based on the analysis results received from the emotion engine, the server generates video and audio for a specific person to respond according to the user's emotions. Analyzed emotion data and a trained model are used as input. Specifically, a prompt sentence is input into the generative AI model to generate video and audio. The generated video and audio data is obtained as output.

[0432] Step 6: Providing a response

[0433] The server transmits the generated video and audio to the terminal. The generated video and audio data is used as input. Specifically, the data is transmitted to the terminal via a network, and the terminal provides the received video and audio to the user. The video and audio viewed by the user are obtained as output.

[0434] (Application example 1)

[0435] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0436] Conventional systems were able to learn the words, actions, and guidelines of a specific person and generate video and audio that sounded like that person was speaking in response to a user's questions or concerns. However, they were unable to respond appropriately according to the user's emotions, and improving the user experience was an issue. In addition, since the generation of prompt sentences using a generative AI model was not performed, more advanced responses based on information about specific people were required.

[0437] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0438] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question or consultation, a means for providing the generated video / audio internally or externally, a means for generating video / audio according to the user's emotions using an emotion engine that recognizes the user's emotions, and a means for generating a prompt sentence based on information about the specific person using a generative AI model. This enables an appropriate response according to the user's emotions and realizes an advanced response based on information about the specific person.

[0439] A "specific person" is an individual whose words, actions, code of conduct, and life philosophy are to be studied based on publicly available information.

[0440] "Speech and behavior" refers to the words and actions of a particular person.

[0441] A "code of conduct" refers to the basic ideas or policies that a particular person follows when acting.

[0442] A "philosophy of life" refers to a person's basic thoughts and values ​​regarding life.

[0443] "Past books" refer to books written by a particular person or books related to that person.

[0444] "Means of learning" refers to a method or device for learning a specific person's words and actions, code of conduct, life philosophy, past books, etc. using machine learning technology.

[0445] "Questions and inquiries" refer to inquiries or requests for advice that users input into the system.

[0446] "Means for generating video / audio" refers to a method or device for generating video and audio that sounds like a specific trained person is speaking.

[0447] The "means for providing" refers to a method or device for providing the generated video and audio internally or externally.

[0448] An "emotion engine" is a technology or device that recognizes a user's emotions and responds according to those emotions.

[0449] A "generative AI model" is an artificial intelligence model that generates prompt sentences based on information about a specific person.

[0450] A "prompt sentence" is a sentence that is generated using a generative AI model based on information about a specific person.

[0451] A system for implementing this invention includes means for learning the words and actions, code of conduct, philosophy of life, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person is speaking in response to an input question, consultation, etc., means for providing the generated video / audio internally or externally, means for generating video / audio in accordance with the user's emotions using an emotion engine that recognizes the user's emotions, and means for generating prompt sentences based on information about a specific person using a generative AI model.

[0452] Explanation of system program processing

[0453] The server retrieves data from the Internet to collect public information about a specific person. The collected data is trained using a deep learning model (e.g., TensorFlow or PyTorch). The trained model is used to understand the specific person's behavior, course of action, and life philosophy.

[0454] When a user inputs a question or inquiry into the system, the server uses an emotion engine to recognize the user's emotions. Based on the recognized emotions, a prompt sentence is generated using a generative AI model (e.g., GPT-3). This prompt sentence is based on information about a specific person and is intended to generate an appropriate response according to the user's emotions.

[0455] Based on the generated prompt, the server generates video and audio that sounds like a specific person is speaking. The generated video and audio are provided to the user via their smartphone or a website.

[0456] Examples

[0457] For example, consider the case of collecting information on famous authors and generating content that would comfort a user when the author is sad. The server collects public information on authors and learns that information using a deep learning model. When a user enters "I've recently failed at work and I'm feeling depressed," the emotion engine recognizes the user's sadness. The generative AI model generates a prompt sentence like the following:

[0458] Collect public information about famous authors and generate content that will comfort your users when they are sad.

[0459] Based on this prompt, the server generates images and sounds that are comforting to the user, allowing the user to receive an appropriate response from a specific person and obtain emotional support.

[0460] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0461] Step 1:

[0462] The server collects public information about a particular person.

[0463] Input: Name of a specific person

[0464] Specific operation: The server uses an API on the Internet to retrieve public information such as articles, books, videos, and audio related to a particular person.

[0465] Output: A dataset of collected public information

[0466] Step 2:

[0467] The server trains a deep learning model based on the collected public information.

[0468] Input: A dataset of collected public information

[0469] How it works: The server uses deep learning frameworks such as TensorFlow and PyTorch to train a model that learns a particular person's words, actions, code of conduct, and life philosophy.

[0470] Output: A trained deep learning model

[0471] Step 3:

[0472] A user inputs a question or inquiry into the system.

[0473] Input: The text of the user's question or request

[0474] Specific actions: The user inputs a question or inquiry through a smartphone or website interface.

[0475] Output: Text data of user questions and inquiries

[0476] Step 4:

[0477] The server recognizes the user's emotions using an emotion engine.

[0478] Input: Text data of user questions and inquiries

[0479] Specific operation: The server uses the EmotionEngine to analyze emotions from the user's text data.

[0480] Output: User emotion data (e.g. happy, sad, angry, etc.)

[0481] Step 5:

[0482] The server generates a prompt sentence using a generative AI model.

[0483] Input: User emotion data, information about specific people

[0484] Specific operation: The server uses a generative AI model such as GPT-3 to generate prompt sentences that correspond to the user's emotions based on information about a specific person.

[0485] Output: Generated prompt statement

[0486] Step 6:

[0487] The server generates video and audio that sounds like a specific person is speaking based on the prompt text.

[0488] Input: Generated prompt sentence, trained deep learning model

[0489] Specific operation: The server uses the generated prompt text to run an algorithm to generate video and audio that sounds like a specific person is speaking.

[0490] Output: The generated video and audio data

[0491] Step 7:

[0492] The server provides the generated video and audio to the user.

[0493] Input: Generated video and audio data

[0494] Specific operation: The server distributes the generated video and audio to the user via a smartphone or website.

[0495] Output: Video and audio data that can be viewed by the user

[0496] Example 2

[0497] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0498] Conventional systems could generate video / audio that sounded as if a specific person was answering a user's questions or concerns, but they could not change the content according to the user's emotions. This made it difficult to provide an appropriate answer that was in line with the user's emotions. In addition, there were limited means to effectively provide the generated video / audio.

[0499] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0500] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question or inquiry, a means for analyzing the user's emotions, a means for changing the content of the generated video / audio based on the analyzed emotions, and a means for providing the generated video / audio internally or externally. This makes it possible to provide an appropriate answer according to the user's emotions and to effectively provide the generated video / audio.

[0501] A "specific person" is an individual that the system learns from, and is the subject of video / audio generation based on that individual's words and actions, code of conduct, philosophy of life, past books, etc.

[0502] "Behavior" refers to what a particular person says or does, and is part of the data that the system learns.

[0503] "Code of conduct" refers to the principles or policies of behavior that a particular person should follow, and is part of the data that the system learns.

[0504] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life, and is part of the data that the system learns.

[0505] "Past books" refers to books written by or about a particular person, and are part of the data the system learns from.

[0506] "Means of learning" refers to techniques and methods for importing a particular person's words and actions, code of conduct, philosophy of life, past books, etc. into the system and analyzing them.

[0507] "Input questions and inquiries" refers to questions and inquiries submitted by users to the system.

[0508] "Means for generating video / audio" refers to technology or methods for creating video and audio that sounds as if a specific trained person is answering an input question or inquiry.

[0509] "Means for analyzing user emotions" refers to techniques and methods for reading and analyzing emotions from user input data.

[0510] "Means for changing the content of generated video / audio" refers to techniques or methods for adjusting the content of generated video and audio based on the analyzed emotions of the user.

[0511] "Means of delivery" refers to the technology or methods used to deliver the generated video / audio to users or other recipients.

[0512] "Providing internally or externally" refers to distributing the generated video / audio to company members or external users.

[0513] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0514] The present invention is a system that generates video / audio that sounds like a specific person is speaking and changes the content according to the user's emotions. A specific embodiment of this system is described below.

[0515] System Configuration

[0516] This system consists of three main elements: a server, a terminal, and a user. The server runs a program that combines natural language processing technology and audio / video generation technology. The terminal is a device through which users input their questions or inquiries, and can include PCs, smartphones, and tablets. The user is an individual who uses the system to ask questions or make inquiries.

[0517] Hardware and Software Used

[0518] The server is a computer equipped with a high-performance processor and a large amount of memory, and uses the following software:

[0519] Natural language processing models: Google's BERT, OpenAI's GPT-4

[0520] Speech generation technology: Amazon Polly, Google Text-to-Speech

[0521] Video generation technology: Deepfake technology, GAN (Generative Adversarial Network)

[0522] Sentiment analysis technology: Microsoft® Azure® Emotion API, Google Cloud's Natural Language API

[0523] System Operation

[0524] The user uses a terminal to input a question or inquiry into the system's interface, for example through a form on a web browser or a dedicated application. The input data is then sent to the server.

[0525] The server analyzes the received text data of the question or inquiry using a natural language processing model (e.g., Google's BERT or OpenAI's GPT-4). This analysis allows the intent and content of the question or inquiry to be understood.

[0526] Next, the server analyzes the user's input data to determine emotions. For emotion analysis, it uses Microsoft Azure's Emotion API and Google Cloud's Natural Language API. Based on the analysis results, it identifies the user's emotions, such as whether they are happy, sad, or angry.

[0527] The server generates video / audio that sounds like a specific person is answering based on the analyzed question or consultation content and the user's emotions. Amazon Polly and Google Text-to-Speech are used for voice generation, and Deepfake technology and GAN are used for video generation. For example, if a user enters "I would like some advice on a new project" and emotion analysis determines that the user is "excited," the server generates video / audio that sounds like a specific business consultant is sharing his / her excitement.

[0528] The generated video / audio is provided to users by the server. It is provided to company members via email or the company network, and is made available on websites via streaming services such as YouTube (registered trademark) and Vimeo, or by providing a direct download link. For example, the link to the generated video / audio is sent by email or uploaded to an in-house portal site.

[0529] Examples and prompts

[0530] As a concrete example, consider a case where a user asks, "I would like some advice on a new project." The server analyzes the question and generates video and audio that sounds as if a specific person (e.g., a well-known business consultant) is answering the question.

[0531] Example prompt:

[0532] "I'd like some advice on a new project. Can you generate video / audio of a specific person answering?"

[0533] In this way, users can have the experience of feeling as if a specific person is actually answering their questions or inquiries.

[0534] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0535] Step 1:

[0536] The user inputs a question or inquiry.

[0537] A user uses a terminal (PC, smartphone, tablet, etc.) to input a question or request for advice into the system's interface. For example, the user might enter "I'd like some advice on a new project" through a form on a web browser or a dedicated application, and then click the send button. The input data is sent to the server in text format.

[0538] Step 2:

[0539] The server receives the input data and performs natural language processing.

[0540] The server receives text data of questions or inquiries sent by users. The received data is analyzed using a natural language processing model (e.g., Google's BERT or OpenAI's GPT-4). Through this analysis, the intent and content of the question or inquiry are understood. The input is the user's text data, and the output is data containing the analyzed intent and content.

[0541] Step 3:

[0542] The server analyzes the user's emotions.

[0543] The server analyzes emotions from the user's input data. For emotion analysis, it uses Microsoft Azure's Emotion API and Google Cloud's Natural Language API. Based on the analysis results, it identifies the user's emotions, such as whether they are happy, sad, or angry. The input is the user's text data, and the output is the analyzed emotion data.

[0544] Step 4:

[0545] The server generates video / audio of a specific person.

[0546] The server generates video / audio that sounds as if a specific person is answering based on the analyzed question or consultation content and the user's emotions. Amazon Polly or Google Text-to-Speech is used for audio generation, and Deepfake technology or GAN (Generative Adversarial Network) is used for video generation. For example, if a user enters "I would like some advice about a new project" and emotion analysis determines that the user is "excited," video / audio is generated in which a specific business consultant shares their excitement. The input is the analyzed intent and emotion data, and the output is the generated video / audio data.

[0547] Step 5:

[0548] The server provides the generated video / audio.

[0549] The server provides the generated video / audio to users. It provides it to internal members via email or the internal network, and makes it available on websites via streaming services such as YouTube or Vimeo, or by providing a direct download link. For example, the link to the generated video / audio is sent by email or uploaded to an internal portal site. The input is the generated video / audio data, and the output is the link or file provided to the user.

[0550] (Application example 2)

[0551] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0552] Conventional Q&A systems have difficulty in providing personalized answers according to the user's emotions, limiting the user experience. In addition, there is a lack of technology to generate realistic images and audio that make it seem as if a specific person is answering. This makes it difficult for users to obtain the specific, emotionally sensitive answers they desire.

[0553] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0554] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person is speaking in response to an input question / consultation, a means for recognizing the user's emotions, a means for adjusting the content and tone of the video / audio generated according to the user's emotions, and a means for providing the generated video / audio internally or externally. This makes it possible to provide a personalized answer according to the user's emotions and generate realistic video and audio that sounds like a specific person is answering.

[0555] A "specific person" is an individual, such as a celebrity or an expert, who is set as a respondent when a user asks a question or makes a request for advice.

[0556] "Words and actions" are records of words and actions that a particular person has made in the past.

[0557] A "code of conduct" is a principle or policy of behavior that a particular person should follow.

[0558] A "philosophy of life" is a person's way of thinking and values ​​regarding life.

[0559] "Past books" are books or publications written by a particular person.

[0560] "Means of learning" refers to technology for collecting and analyzing data such as a specific person's words and actions, code of conduct, philosophy of life, past books, etc.

[0561] "Questions and inquiries" are questions or concerns that users input into the system.

[0562] "Means for generating video / audio" refers to technology that creates video and audio that sounds as if a specific person is actually speaking.

[0563] "Means of recognizing emotions" refers to technology that analyzes emotions from a user's facial expressions and voice.

[0564] "Means for adjusting content and tone" refers to technology that changes the content and tone of generated video and audio in response to the user's emotions.

[0565] "Means of providing" refers to the technology used to deliver the generated video and audio to the user.

[0566] "Internal or external" refers to users within the system or external websites.

[0567] A system for implementing this invention includes means for learning the words and actions, code of conduct, philosophy of life, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person learned is speaking in response to an input question / consultation, etc., means for recognizing a user's emotions, means for adjusting the content and tone of the generated video / audio in response to the user's emotions, and means for providing the generated video / audio internally or externally.

[0568] System Configuration

[0569] The server uses the following hardware and software:

[0570] Hardware: PC with high-performance CPU and GPU, web camera

[0571] Software: Python, OpenAI API, OpenCV, EmotionRecognizer (emotion recognition library)

[0572] Data processing and calculation

[0573] 1. Entering a question: The user enters a question or inquiry through the terminal.

[0574] 2. Emotion Recognition: The server captures the user's facial expressions using a webcam and recognizes the emotions using the EmotionRecognizer library.

[0575] 3. Answer Generation: The server uses the OpenAI API to generate an answer based on the question and the user's sentiment. Specifically, it generates a prompt sentence like the following:

[0576] Example prompt:

[0577] User Question: I've been having trouble at work lately. What should I do?

[0578] User Emotion: Sadness

[0579] Generate content for a specific person to answer.

[0580] 4. Video Generation: The server uses OpenCV to generate video frames containing the generated answer text and saves them as video files.

[0581] 5. Provision: The server provides the generated video and audio to the user in streaming or download format.

[0582] Examples

[0583] If a user inputs the question "I'm worried that things haven't been going well at work lately. What should I do?" and the emotion recognition system recognizes the user's emotion as "sadness," the server will generate the following prompt sentence:

[0584] User Question: I've been having trouble at work lately. What should I do?

[0585] User Emotion: Sadness

[0586] Generate content for a specific person to answer.

[0587] Based on this prompt, the OpenAI API is used to generate a response that sounds like a specific person is answering, and a video containing that response is generated. The generated video is provided with content and tone that is in tune with the user's emotions.

[0588] The flow of the identification process in the application example 2 will be described with reference to FIG.

[0589] Step 1:

[0590] The user inputs questions or inquiries through the terminal.

[0591] Input: Text data of user's question or inquiry

[0592] Output: The text data of the question or inquiry is sent to the server.

[0593] Specific operation: The user enters a question or inquiry into the input form on the terminal and presses the send button. The terminal sends the entered text data to the server.

[0594] Step 2:

[0595] The server captures the user's facial expressions using a webcam and recognizes emotions using the EmotionRecognizer library.

[0596] Input: Video data of the user's facial expression

[0597] Output: User emotion data (e.g. happy, sad, angry, etc.)

[0598] Specific operation: The server starts the webcam and captures the user's facial expressions in real time. The captured video data is input to the EmotionRecognizer library and the emotions are analyzed. Emotion data is obtained as the analysis result.

[0599] Step 3:

[0600] The server uses the OpenAI API to generate answers based on the question and the user's sentiment.

[0601] Input: Text data of questions and inquiries, user emotion data

[0602] Output: Answer text data

[0603] Specific operation: The server generates a prompt sentence by combining the text data of the question or inquiry with the user's emotion data. The generated prompt sentence is sent to the OpenAI API, and an answer is generated as if a specific person were answering. The text data of the generated answer is obtained.

[0604] Step 4:

[0605] The server uses OpenCV to generate video frames containing the generated answer text and saves them as video files.

[0606] Input: Text data of the answer

[0607] Output: Video file (movie file)

[0608] What happens: The server uses OpenCV to generate video frames that show the answer text, and then saves the frames in sequence as a video file.

[0609] Step 5:

[0610] The video and audio generated by the server are provided to the user in streaming or download format.

[0611] Input: Video file (movie file)

[0612] Output: Video data that can be viewed by the user

[0613] Specific operation: The server uploads the generated video file to the streaming server and generates a URL that users can access. Users can watch the video through the URL. Also, the server provides the video file in a download format if necessary.

[0614] Example 3

[0615] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0616] Conventional systems could generate video and audio by utilizing the knowledge and experience of a specific person, but it was difficult to respond appropriately to the user's emotions. In addition, there was a problem that the quality of the user experience was reduced because the system could not respond to changes in the user's emotions in real time. Furthermore, there were limited means to effectively provide the generated video and audio, so knowledge and experience were not shared sufficiently.

[0617] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0618] In this invention, the server includes a means for learning the words and actions, guidelines for action, philosophy of life, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question / consultation, etc., a means for collecting user emotion data, a means for analyzing the collected emotion data, a means for generating a prompt sentence based on the analysis result, a means for generating video / audio based on the generated prompt sentence, and a means for providing the generated video / audio internally or externally. This makes it possible to generate and provide appropriate video and audio according to the user's emotions, and to effectively utilize and widely share the knowledge and experience of a specific person.

[0619] A "specific person" refers to an individual with specific knowledge or experience.

[0620] "Behavior" refers to the words and actions of a particular person.

[0621] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[0622] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[0623] "Past books" refers to books and publications written by a specific person.

[0624] "Means of learning" refers to a method of inputting a specific person's words and actions, code of conduct, philosophy of life, past books, etc. into the system and having it learn from them.

[0625] "Questions and inquiries" refers to inquiries or requests for advice that users input into the system.

[0626] "Means for generating video / audio" refers to a method for producing video or audio based on learned content.

[0627] "Emotion data" refers to information about emotions derived from a user's facial expressions, tone of voice, etc.

[0628] "Means of collection" refers to methods of obtaining emotional data using devices such as cameras and microphones.

[0629] "Means for analyzing" refers to a method for analyzing collected emotion data to recognize a user's emotional state.

[0630] A "prompt sentence" refers to an instruction sentence to be input into a generative AI model.

[0631] "Means of generation" refers to the method of producing images and sounds based on the prompt text.

[0632] "Means of providing" refers to the method of distributing the generated video and audio internally or externally.

[0633] "Internal" refers to users and members within the system.

[0634] "External" refers to public access or websites outside the system.

[0635] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0636] This invention is a system that utilizes the knowledge and experience of a specific person to generate and provide video and audio corresponding to the user's emotions. A specific embodiment of this system is described below.

[0637] System Configuration

[0638] The system is composed of three main elements: a server, a terminal, and a user. The server includes a means for learning a specific person's words and actions, guidelines for action, life philosophy, past books, etc., a means for analyzing emotion data, a means for generating prompts, a means for generating video / audio, and a means for providing the generated video / audio. The terminal includes a means for collecting user emotion data.

[0639] Hardware and Software Used

[0640] The server is a computer system equipped with a high-performance processor and large-capacity memory. Specifically, it uses NVIDIA's GPU to perform high-speed analysis of emotion data. OpenAI's GPT-4 is used as the generative AI model. The terminal is a device equipped with a camera and microphone that collects the user's facial expressions and tone of voice in real time.

[0641] Data processing and calculation

[0642] The device collects the user's facial expressions and tone of voice through a camera and microphone. The collected data is sent to a server. The server analyzes the data using an emotion engine to recognize the user's emotional state. Based on the analysis results, the server generates a prompt sentence to be input into a generative AI model. The generative AI model generates video and audio based on the prompt sentence. The generated video and audio are provided internally or externally through the server.

[0643] Examples

[0644] For example, if a user is confused during a presentation, the device collects the user's confused facial expression and trembling voice. The server uses an emotion engine to recognize that the user is confused and generates a prompt sentence such as "The user is confused, so please generate an encouraging message." Based on this prompt sentence, the generative AI model generates an encouraging message such as "It's okay, let's move on to the next question." The generated message is displayed in real time on the user's device to support the user.

[0645] Example prompt:

[0646] The user is confused, so generate an encouraging message.

[0647] In this way, a system is realized in which the server, the terminal, and the user work together to generate and provide appropriate video and audio in accordance with the user's emotions. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0648] Step 1:

[0649] The device collects the user's emotional data. Specifically, it captures the user's facial expressions with a camera and records the tone of voice with a microphone. This allows the user's facial movements and voice changes to be acquired as data in real time. The input is the user's facial expressions and voice, and the output is the collected emotional data.

[0650] Step 2:

[0651] The terminal transmits the collected emotion data to the server. Specifically, the terminal transfers the collected emotion data to the server through a network. The input is the collected emotion data, and the output is the emotion data transmitted to the server.

[0652] Step 3:

[0653] The server analyzes the emotion data. Specifically, it uses an emotion engine to analyze the data and recognize the user's emotional state. For example, if the user's eyebrows are furrowed or their voice is trembling, the emotion engine recognizes that the user is confused. The input is the emotion data sent to the server and the output is the analyzed emotional state.

[0654] Step 4:

[0655] The server generates a prompt sentence based on the analysis result. Specifically, it generates a prompt sentence such as "The user is confused, so please generate an encouraging message" based on the emotional state obtained from the emotion engine. The input is the analyzed emotional state, and the output is the generated prompt sentence.

[0656] Step 5:

[0657] The server inputs a prompt sentence into the generative AI model and generates video and audio. Specifically, the server inputs a prompt sentence into a generative AI model (for example, OpenAI's GPT-4) and generates video and audio according to the user's emotions. For example, it generates an encouraging message such as "It's okay, let's move on to the next question." The input is the generated prompt sentence, and the output is the generated video and audio.

[0658] Step 6:

[0659] The server provides the generated video and audio. Specifically, the generated video and audio are provided to company members or published on a website. For example, a generated message of encouragement is displayed on the user's terminal in real time to support the user. The input is the generated video and audio, and the output is the provided video and audio.

[0660] (Application example 3)

[0661] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0662] Conventional systems can generate video and audio to utilize the knowledge and experience of a specific person, but they have the problem of being unable to respond appropriately to changes in the user's emotions. This means that the system is unable to respond appropriately to the user's emotions, resulting in a lack of improvement in the user experience.

[0663] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a means for learning the words and actions, guidelines for action, philosophy of life, past books, etc. of a specific person, a means for generating video / audio that sounds like the learned specific person is speaking in response to an input question / consultation, etc., a means for providing the generated video / audio internally or externally, a means for recognizing changes in the user's emotions in real time, a means for generating video / audio in response to changes in emotions, and a means for playing the generated video / audio. This makes it possible to generate and provide appropriate video and audio in real time according to the user's emotions.

[0664] A "specific person" refers to an individual with specific knowledge or experience.

[0665] "Speech and behavior" refers to the words and actions of a particular person.

[0666] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[0667] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[0668] "Past books" refers to books and documents written by a specific person.

[0669] "Means of learning" refers to methods or techniques for making a system learn a particular person's words and actions, code of conduct, philosophy of life, past books, etc.

[0670] "Input questions and inquiries" refers to the content of questions and inquiries that users make to the system.

[0671] "Means for generating video / audio" refers to methods or technologies for producing video or audio based on learned content.

[0672] "Means of providing" refers to the methods and technologies for providing the generated video and audio internally or externally.

[0673] "Means for recognizing changes in emotions in real time" refers to a method or technology for detecting changes in a user's emotions in real time.

[0674] "Means for generating video / audio in response to changes in emotions" refers to methods or technologies for creating appropriate video and audio based on changes in a user's emotions.

[0675] "Video / audio playback means" refers to the methods and techniques for playing the generated video and audio to the user.

[0676] A system for implementing this invention includes means for learning the words and actions, code of conduct, philosophy of life, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person who has been learned is speaking in response to an input question or inquiry, means for providing the generated video / audio internally or externally, means for recognizing changes in a user's emotions in real time, means for generating video / audio in response to changes in emotions, and means for playing back the generated video / audio.

[0677] System Programming

[0678] The program of this system performs the following processes.

[0679] Hardware and Software

[0680] Hardware: Smartphone camera

[0681] Software: OpenCV (face detection), Keras (emotion recognition model), text_to_speech (audio generation), video_generator (video generation)

[0682] Data processing and calculation

[0683] The server captures video in real time using the smartphone camera. It detects faces using OpenCV and converts the detected face images to grayscale. It then predicts emotions from the face images using Keras emotion recognition models. Depending on the predicted emotions, it generates appropriate audio using the text_to_speech library and appropriate video using the video_generator library. The generated video and audio are then played back to the user.

[0684] Examples

[0685] For example, when a user is using the app through their smartphone camera, if the emotion recognition model detects "sadness" from the user's facial expression, the server will generate audio saying, "I see you are feeling sad. Here is something to comfort you," and play a video comforting the user.

[0686] Examples of prompt statements

[0687] "When a user is using the app through the smartphone camera, if the emotion recognition model detects a specific emotion from the user's facial expression, please create an application that generates and plays audio and video corresponding to that emotion."

[0688] In this way, it becomes possible to generate and provide appropriate video and audio in real time according to the user's emotions.

[0689] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0690] Step 1:

[0691] The server captures video in real time using the smartphone camera. The input is video data from the camera, and the output is the captured video frames. Specifically, the camera starts up and continuously captures video including the user's face.

[0692] Step 2:

[0693] The server uses OpenCV to detect faces from the captured video frames. The input is the video frame obtained in step 1, and the output is the area information (coordinates) of the detected face. The specific operation is to convert the video frame to grayscale and apply a face detection algorithm to identify the position of the face.

[0694] Step 3:

[0695] The server cuts out the detected face area and converts it into a grayscale image. The input is the face area information obtained in step 2 and the original video frame, and the output is a grayscale image of the face. The specific operation is to cut out the face area and resize it to 48x48 pixels.

[0696] Step 4:

[0697] The server uses a Keras emotion recognition model to predict emotions from grayscale facial images. The input is the grayscale facial image obtained in step 3, and the output is the predicted emotion label. The specific operation is to input the facial image into the model, calculate the probability distribution of emotions, and select the emotion label with the highest probability.

[0698] Step 5:

[0699] The server generates an appropriate speech according to the predicted emotion using the text_to_speech library. The input is the emotion label obtained in step 4, and the output is the generated speech data. The specific operation is to select a text message corresponding to the emotion label and convert the text to speech.

[0700] Step 6:

[0701] The server generates an appropriate video according to the predicted emotion using the video_generator library. The input is the emotion label obtained in step 4, and the output is the generated video data. Specifically, the server selects a video template corresponding to the emotion label and generates a video based on the template.

[0702] Step 7:

[0703] The server plays the generated video and audio to the user. The input is the audio data obtained in step 5 and the video data obtained in step 6, and the output is the video and audio played to the user. In concrete terms, the video and audio are played simultaneously using the smartphone's display and speaker.

[0704] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0705] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0706] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0707] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0708] [Second embodiment]

[0709] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0710] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0711] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0712] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0713] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0714] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0715] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0716] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0717] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0718] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0719] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0720] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0721] "Example 1"

[0722] As one embodiment of the present invention, there is a system equipped with a means for automatically collecting public information of a specific person, such as articles, books, videos, audio, etc. on the Internet. This system learns the words and actions, guidelines for action, and life philosophy of the specific person based on the collected information. Machine learning techniques such as deep learning are used for the learning.

[0723] "Example 2"

[0724] Furthermore, as one embodiment of the present invention, there is a system equipped with a means for generating video / audio that sounds as if a specific person who has been trained is speaking in response to an input question / consultation, etc. This system combines natural language processing technology and audio / video generation technology to generate video / audio that sounds as if a specific person is actually answering the question / consultation.

[0725] "Example 3"

[0726] As one embodiment of the present invention, there is a system equipped with a means for providing the generated video / audio internally or externally. This system makes it possible to utilize the knowledge and experience of a specific person by providing the generated video / audio to members of the company. In addition, by publishing the generated video / audio on a website, it becomes possible to share that knowledge and experience widely.

[0727] The process flow of each embodiment will be described below.

[0728] "Example 1"

[0729] Step 1: Automatically collect public information about a specific person, such as articles, books, videos, audio, etc. on the Internet. This collection is done using web scraping technology and APIs.

[0730] Step 2: Based on the collected information, the behavior, code of conduct, and life philosophy of a specific person are learned. This learning is done using machine learning techniques such as deep learning. Specifically, a neural network is trained to generate a model that mimics the behavioral patterns of a specific person.

[0731] "Example 2"

[0732] Step 1: Receive input from the user, such as a question or inquiry. This input can be in text or voice format.

[0733] Step 2: In response to the input question or inquiry, video / audio is generated that sounds as if a specific person who has been trained is speaking. This generation is performed by combining natural language processing technology and audio / video generation technology. Specifically, using the trained model, video / audio is generated that sounds as if a specific person is actually answering the question or inquiry.

[0734] "Example 3"

[0735] Step 1: The generated video / audio is provided internally or externally. This provision can take various forms, such as sharing information with company members or publishing on a website. Specifically, for company members, it is provided via email or the company network, and for publishing on a website, it is provided as a streaming service or download.

[0736] Example 1

[0737] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0738] In conventional systems, data collection and preprocessing to learn a specific person's words and actions, guidelines for action, and life philosophy were often done manually, which was problematic in terms of efficiency. In addition, the performance evaluation of the trained model and the generation of responses to user questions were not sufficiently automated, resulting in a lack of accuracy and reliability. Furthermore, there were limited means to effectively provide the generated video and audio, which made it difficult to improve the user experience.

[0739] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0740] In this invention, the server includes a means for automatically collecting public information related to a specific person from the Internet, a means for preprocessing the collected data, a means for training a deep learning model using the preprocessed data, a means for evaluating the performance of the trained model, and a means for generating responses to questions from users. This makes it possible to automate a series of processes from data collection to model training, performance evaluation, and response generation, and provide an efficient and highly accurate system.

[0741] A "specific person" refers to a specific individual, and refers to information regarding their words and actions, code of conduct, life philosophy, past works, etc.

[0742] "Behavior" refers to words and actions made by a particular person, and reflects that person's characteristics and personality.

[0743] A "code of conduct" refers to the basic policies or principles by which a particular person acts, and indicates that person's values ​​and beliefs.

[0744] A "philosophy of life" refers to a person's thoughts and beliefs about life, and reflects that person's way of life and values.

[0745] "Past books" refer to books written by a particular person or about that person, which record that person's thoughts and activities.

[0746] "Public information" refers to information available on the Internet or from other public sources, including data relating to a particular person.

[0747] "Automated collection means" refers to the use of programs or algorithms to collect publicly available information relating to a particular person without manual intervention.

[0748] "Preprocessing means" refers to processes for converting collected data into a format that is easier to analyze, including data cleaning and normalization.

[0749] A "deep learning model" refers to a machine learning model that uses neural networks and has the ability to learn complex patterns and features.

[0750] "Means of evaluating performance" refers to methods for measuring the accuracy and reliability of a trained model, using evaluation metrics and test data.

[0751] The "means for generating a response" refers to a method for generating an appropriate answer to a question from a user, and creates a response using a learned model.

[0752] "Means for generating video / audio" refers to technology for creating video and audio that sounds like a specific person is speaking, using a generative AI model.

[0753] "Means of providing" refers to the method for providing the generated video and audio to the user, which is carried out through an internal system or an external platform.

[0754] "Means for publishing on a website" refers to a method for publishing the generated video and audio on a website on the Internet, providing them in a format that can be accessed by users.

[0755] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0756] This invention relates to a system that learns the words and actions, guidelines, life philosophies, past books, etc. of a specific person. This system automatically collects public information related to a specific person, preprocesses the collected data, and trains it using a deep learning model. Furthermore, the system evaluates the performance of the trained model and generates responses to questions from users.

[0757] Data collection

[0758] The server automatically collects public information related to a specific person from the Internet. Specifically, it uses Python's BeautifulSoup and Scrapy to scrape web pages and extract the necessary information. For example, it collects data such as articles, books, videos, and audio related to a specific person.

[0759] Data Preprocessing

[0760] The server preprocesses the collected data: for text data, it uses NLTK or spaCy to clean, tokenize, and normalize the text, and for audio data, it uses the Google Cloud Speech-to-Text API to convert the audio data to text, which puts the data in a format that's easier to parse.

[0761] Model Training

[0762] The server trains a deep learning model using the preprocessed data. It uses TensorFlow and PyTorch to build a model that learns a specific person's words and actions, behavioral guidelines, and life philosophy. Specifically, it trains models such as LSTM and Transformer using the collected text data as input.

[0763] Model evaluation

[0764] The server evaluates the performance of the trained model using metrics such as precision, recall, and F1 score. Specifically, it evaluates the model's predictive performance using test data and adjusts the model's hyperparameters as necessary.

[0765] Generate a response to the user

[0766] The user inputs a question about a specific person into the generative AI model from the terminal. The server receives the prompt sentence from the user and generates a response using the trained model. For example, if the user inputs "Tell me about the leadership style of a specific person," the server uses the model to generate information about the person's leadership style and provides it to the user.

[0767] Examples of concrete examples and prompts

[0768] For example, if a specific person is a "famous leader," the server will collect articles, books, videos, and audio about that leader from the Internet. The collected data will be preprocessed and the deep learning model will be used to learn the leader's words, actions, and life philosophy.

[0769] Examples of prompts to be input to a generative AI model include:

[0770] "Tell me about the leadership styles of some famous leaders."

[0771] "Describe how a notable leader led themselves to success."

[0772] "Tell me more about the life philosophies of famous leaders."

[0773] In this way, a user can obtain detailed information about a particular person.

[0774] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0775] Program processing flow

[0776] Step 1: Data collection

[0777] The server automatically collects public information related to a specific person from the Internet. Specifically, it uses Python's BeautifulSoup and Scrapy to scrape web pages and extract the necessary information. As input, it receives a list of URLs related to a specific person, and as output, it generates the collected text data and metadata.

[0778] Step 2: Data Preprocessing

[0779] The server preprocesses the collected data. For text data, it uses NLTK or spaCy to clean, tokenize, and normalize the text. For audio data, it uses Google Cloud Speech-to-Text API to convert the audio data to text. As input, it takes the collected raw data and produces preprocessed, clean text data as output.

[0780] Step 3: Model training

[0781] The server trains a deep learning model using the preprocessed data. It uses TensorFlow and PyTorch to build a model that learns a specific person's words and actions, behavioral guidelines, and life philosophy. Specifically, it trains models such as LSTM and Transformer. It receives preprocessed text data as input and generates a trained model as output.

[0782] Step 4: Model evaluation

[0783] The server evaluates the performance of the trained model. For the evaluation, it uses metrics such as precision, recall, and F1 score. Specifically, it evaluates the predictive performance of the model using test data and adjusts the model's hyperparameters as necessary. It receives the trained model and test data as input and generates the evaluation results as output.

[0784] Step 5: Accepting user questions

[0785] A user inputs a question about a specific person into the generative AI model from a terminal. The model receives the user's question as input and sends the question to the server as output.

[0786] Step 6: Response Generation

[0787] The server receives a prompt from the user and generates a response using the trained model. Specifically, the server inputs the user's question into the model and generates an appropriate answer. The server receives the user's question and the trained model as input, and generates the generated response as output.

[0788] Step 7: Providing a response

[0789] The server provides the generated response sentence to the user. Specifically, the server transmits the generated response sentence to the terminal so that the user can view it. The server receives the generated response sentence as input and displays the response sentence on the user's terminal as output.

[0790] In this way, the entire process from data collection to response generation is automated, resulting in an efficient and highly accurate system.

[0791] (Application example 1)

[0792] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0793] In modern society, there is a demand to learn the words and actions, guidelines for action, and life philosophies of specific people and apply them to daily life and work. However, there is a lack of efficient means to collect this information and provide it to users in an appropriate format. Furthermore, there is no system that automatically collects public information about specific people and uses it to provide specific advice and motivation to users. As a result, users must spend a great deal of time and effort collecting and analyzing information.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0795] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question or inquiry, a means for providing the generated video / audio internally or externally, a means for automatically collecting public information about the specific person, and a means for providing the user with advice and motivation that is useful in daily life and work based on the collected information. This allows the user to efficiently learn the philosophy and guidelines of a specific person and use them in daily life and work.

[0796] A "specific person" refers to a specific individual, and refers to information about that person's words and actions, code of conduct, life philosophy, past books, etc.

[0797] "Words and actions" refers to what a particular person says or does.

[0798] A "code of conduct" refers to the basic ideas or policies that a particular person follows when acting.

[0799] "Philosophy of life" refers to a person's basic thoughts and values ​​regarding life.

[0800] "Past books" refers to books written by a particular person or books about that person.

[0801] "Means of learning" refers to a method of learning using machine learning technology based on information such as a specific person's words and actions, code of conduct, philosophy of life, and past books.

[0802] "Question / Inquiry" refers to a question or inquiry entered by a user for a specific person.

[0803] "Means for generating video / audio" refers to means for generating video and audio that sounds like a specific trained person is speaking.

[0804] The "means for providing" refers to the means for providing the generated video and audio internally or externally.

[0805] "Means of automatically collecting public information" refers to means of automatically collecting public information such as articles, books, videos, and audio on the Internet.

[0806] "Means of providing advice and motivation" refers to means of providing users with advice and motivation that is useful in their daily lives and work, based on the collected information.

[0807] A system for implementing this invention includes means for learning the words and actions, guidelines for action, life philosophy, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person is speaking in response to an input question or inquiry, means for providing the generated video / audio internally or externally, means for automatically collecting public information about the specific person, and means for providing the user with advice and motivation that is useful in their daily life and work based on the collected information.

[0808] System Configuration

[0809] Hardware

[0810] Server: A central processing unit for collecting, learning, generating, and providing data.

[0811] Smartphone: A device for users to input questions and inquiries and receive the generated video / audio

[0812] software

[0813] Python: A programming language for data collection, machine learning, and generative processing

[0814] BeautifulSoup: A library for scraping articles and information from the internet

[0815] TensorFlow: A library for building and training deep learning models

[0816] Data processing and calculation

[0817] Data collection

[0818] The server automatically collects public information about a particular person by using BeautifulSoup to scrape information such as articles, books, videos, and audio from the Internet.

[0819] Data Preparation

[0820] Based on the collected information, we create a dataset to learn the words and actions, behavioral guidelines, and life philosophies of a particular person, which includes preprocessing and labeling the text data.

[0821] Model Building and Training

[0822] Using TensorFlow, a Long Short-Term Memory (LSTM) model is built and trained on the collected data, creating a generative AI model that mimics the behavior and course of action of a specific person.

[0823] Video / audio generation

[0824] Using the learned model, video / audio that sounds like a specific person is speaking in response to a question or inquiry entered by the user is generated.

[0825] offer

[0826] The generated video and audio are sent from the server to the user's smartphone, allowing the user to efficiently learn the philosophy and code of conduct of a particular person and apply it to their daily life and work.

[0827] Examples

[0828] When a user inputs the name of a "certain celebrity" into their smartphone, the server collects publicly available information about that person and learns from it. Then, when the user inputs a question such as "What would a particular celebrity think in this situation?", the server uses the learned model to generate video and audio that sounds like the person is speaking and sends it to the smartphone.

[0829] Examples of prompt statements

[0830] Enter the name of a specific celebrity. For example, "Specific Celebrity A" or "Specific Celebrity B".

[0831] In this way, users can learn the philosophies and codes of conduct of a particular person and apply them in their daily lives and work.

[0832] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0833] Step 1:

[0834] A user inputs the name of a specific person into a smartphone terminal. The input name is sent to a server. The input data is the name of a specific person.

[0835] Step 2:

[0836] The server automatically collects public information such as articles, books, videos, and audio on the Internet based on the input name of a specific person. Specifically, it uses BeautifulSoup to perform web scraping to obtain related text data. The input data is the name of a specific person, and the output data is the collected public information.

[0837] Step 3:

[0838] The server creates a dataset based on the collected public information to learn the words and actions, behavioral guidelines, and life philosophies of a specific person. It preprocesses the text data (tokenization, normalization, labeling, etc.) and formats the learning data. The input data is the collected public information, and the output data is the preprocessed learning data.

[0839] Step 4:

[0840] The server uses TensorFlow to build a Long Short-Term Memory (LSTM) model and trains the model based on preprocessed training data. This creates a generative AI model that mimics the words, actions, and behavior of a specific person. The input data is the preprocessed training data, and the output data is the trained generative AI model.

[0841] Step 5:

[0842] A user inputs a question or inquiry into a smartphone terminal. The input question or inquiry is sent to a server. The input data is the content of the user's question or inquiry.

[0843] Step 6:

[0844] The server uses a trained generative AI model to generate video / audio that sounds like a specific person is speaking in response to a question or inquiry input by a user. The input data is the user's question or inquiry content and the trained generative AI model, and the output data is the generated video / audio.

[0845] Step 7:

[0846] The server transmits the generated video / audio to the smartphone terminal. The user can receive and view the generated video / audio on the smartphone terminal. The input data is the generated video / audio, and the output data is the video / audio displayed on the user's smartphone terminal.

[0847] In this way, users can learn the philosophies and code of conduct of a particular person and apply them to their daily lives and work.

[0848] Example 2

[0849] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0850] In conventional systems, it was difficult to generate video and audio that sounded like a specific person was actually answering a user's question or inquiry. In addition, the quality of the generated video and audio was low, making it impossible to provide a realistic experience to the user. Furthermore, there was a lack of means to learn the words, actions, and guidelines of a specific person, so the generated answers lacked reliability and consistency.

[0851] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0852] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for analyzing input text using natural language processing technology for input questions and inquiries, a means for generating a prompt sentence based on the analysis result, a means for inputting the prompt sentence into a generation AI model and generating an answer text, a means for generating audio in the voice of a specific person using speech synthesis technology from the generated answer text, a means for generating an image of a specific person speaking using video generation technology based on the generated audio, and a means for providing the generated video / audio internally or externally. This makes it possible to generate high-quality video and audio as if a specific person were actually answering questions and inquiries from users, providing a realistic experience.

[0853] A "specific person" is a specific individual designated by the user, and is a person whose words and actions, code of conduct, philosophy of life, past books, etc. are to be studied.

[0854] "Behavior" refers to the words and actions taken by a particular person and reflects that person's characteristics and personality.

[0855] A "code of conduct" is a standard or policy of behavior that a particular person should follow, and indicates that person's values ​​and beliefs.

[0856] A "philosophy of life" is a person's thoughts and beliefs about life, which reflect that person's way of life and values.

[0857] "Past books" refer to books written by a particular person or books related to that person, and are materials for learning about that person's ideas and knowledge.

[0858] "Natural language processing technology" refers to technology that enables computers to understand and analyze human language, analyzing text and extracting meaning.

[0859] A "prompt" is a document to be input into a generative AI model, and is an instruction text for generating answers to users' questions or inquiries.

[0860] A "generative AI model" is a model that uses artificial intelligence to generate text or speech, and generates answers based on prompts.

[0861] "Speech synthesis technology" refers to technology that converts text into speech and is used to generate speech in the voice of a specific person.

[0862] "Image generation technology" refers to technology that generates images based on audio or text, and is used to create images of a specific person speaking.

[0863] "Means for providing internally or externally" refers to means for storing the generated video and audio within the system, or for providing them to an external user or system.

[0864] The present invention is a system for generating video and audio that sounds as if a specific person is actually answering a question or inquiry from a user. This system is implemented using the following hardware and software.

[0865] Hardware and Software Used

[0866] Hardware: Server equipped with a high-performance GPU

[0867] Software: Natural Language Processing (NLP) libraries (e.g. spaCy, Transformers), voice synthesis software (e.g. Tacotron 2), video generation software (e.g. DeepFake technology)

[0868] Specific operation of the system

[0869] When the server receives a question or request for advice from a user, it first analyzes the input text using natural language processing technology. The analysis includes grammatical analysis and keyword extraction. It then generates a prompt sentence based on the analysis results. For example, if a user enters, "I haven't been able to sleep at night lately. What should I do?", the following prompt sentence will be generated:

[0870] Example prompt:

[0871] "Health Advice: I haven't been able to sleep at night lately, what should I do? Let your doctor answer this question."

[0872] The server inputs this prompt sentence into a generative AI model (e.g., GPT-3) to generate an answer text. The generated answer text might look something like this:

[0873] Example of a generated answer:

[0874] "There are several possible reasons why you can't sleep at night. First, it's important to make time to relax before going to bed. For example, taking a warm bath or doing some light stretching is effective. It's also important to limit your caffeine intake."

[0875] Next, the server inputs the generated response text into speech synthesis software to generate audio in a specific person's voice. For example, audio is generated in the voice of a doctor. Then, based on the generated audio, video generation software is used to generate video of the specific person speaking. Finally, the generated audio and video are integrated into a single video file.

[0876] Examples

[0877] For example, if a user enters "I haven't been able to sleep at night lately. What should I do?", the process will proceed as follows:

[0878] 1. User input: The user uses a terminal to enter a question.

[0879] 2. Text analysis: The server uses an NLP library to extract keywords such as "I can't sleep at night" and "What should I do?"

[0880] 3. Prompt generation: The server generates a prompt such as "Health consultation: I haven't been able to sleep at night lately. What should I do? Please have a doctor answer this question."

[0881] 4. Input to the generative AI model: The server inputs the prompt sentence into GPT-3.

[0882] 5. Answer generation: GPT-3 generates the answer, "There are several possible reasons why you can't sleep at night. First, it is important to take time to relax before going to bed. For example, taking a warm bath or doing some light stretching can be effective. It is also important to limit your caffeine intake."

[0883] 6. Speech synthesis: The server uses Tacotron 2 to generate speech in the doctor’s voice.

[0884] 7. Video Generation: The server uses DeepFake technology to generate a video of the doctor speaking.

[0885] 8. Audio and video integration: The server integrates the generated audio and video.

[0886] 9. Result delivery: The server sends the consolidated video file to the user's device, where the user watches the video.

[0887] In this way, the user can experience high quality video and audio as if a specific person were actually answering the question.

[0888] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0889] Step 1:

[0890] Users access the system using a terminal and input their questions or inquiries. The input text is then sent to the server.

[0891] Input: User's question or concern (e.g. "I haven't been able to sleep at night lately. What should I do?")

[0892] Output: The input text sent to the server.

[0893] Step 2:

[0894] The server analyzes the received input text using a natural language processing (NLP) library (e.g., spaCy), which includes grammar analysis and keyword extraction.

[0895] Input: The input text sent by the user

[0896] Data processing: grammatical analysis, keyword extraction

[0897] Output: Analysis results (e.g. keywords such as "I can't sleep at night" and "What should I do")

[0898] Step 3:

[0899] The server generates a prompt sentence based on the analysis results. The prompt sentence is an instruction sentence to be input to the generative AI model.

[0900] Input: Analysis results

[0901] Data processing: Prompt generation

[0902] Output: Prompt text (e.g. "Health question: I've been having trouble sleeping at night lately. What should I do? Please have your doctor answer this question.")

[0903] Step 4:

[0904] The server inputs the generated prompt sentence into a generative AI model (e.g., GPT-3) to generate an answer text.

[0905] Input: prompt statement

[0906] Data computation: Generative AI model generates answers

[0907] Output: Answer text (Example: "There are several possible reasons why you can't sleep at night. First, it's important to take time to relax before going to bed. For example, taking a warm bath or doing some light stretching can be effective. It's also important to limit your caffeine intake.")

[0908] Step 5:

[0909] The server inputs the generated answer text into speech synthesis software (e.g., Tacotron 2) to generate a voice in the voice of a specific person.

[0910] Input: Answer text

[0911] Data Calculation: Voice Synthesis

[0912] Output: Audio data (e.g. generated audio of a doctor's voice)

[0913] Step 6:

[0914] The server uses video generation software (e.g., DeepFake technology) based on the generated audio to generate a video of a specific person speaking.

[0915] Input: Audio data

[0916] Data Calculation: Image Generation

[0917] Output: Video data (e.g. video of a doctor speaking)

[0918] Step 7:

[0919] The server combines the generated audio and video into a single video file.

[0920] Input: Audio data, video data

[0921] Data processing: Audio and video integration

[0922] Output: A merged video file

[0923] Step 8:

[0924] The server transmits the combined video file to the user's terminal, and the user watches the video.

[0925] Input: merged video file

[0926] Output: Video file sent to user's device

[0927] The above is the specific processing flow of the program of this system.

[0928] (Application example 2)

[0929] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0930] In conventional customer service in brick-and-mortar stores, when a customer has a question or inquiry, a sales clerk is required to respond, and because this depends on the number and skills of the sales clerks, the response may be delayed or an insufficient answer may be provided. Furthermore, if a specific sales clerk is not available or there are only a limited number of sales clerks with specialized knowledge, customer satisfaction may decrease. To solve these problems, there is a demand for a system that can generate video and audio that sounds like a specific person is speaking, and have a virtual sales clerk serve customers in a brick-and-mortar store.

[0931] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for learning the words and actions, guidelines for action, philosophy of life, past books, etc. of a specific person, a means for generating video / audio that sounds like the learned specific person is speaking in response to an input question / inquiry, etc., a means for providing the generated video / audio internally or externally, a means for generating video / audio of a virtual store clerk answering a customer's question in a real store, and a means for providing the generated video / audio to a smartphone. This makes it possible for a customer in a real store to get a quick and accurate answer from a virtual store clerk using a smartphone.

[0932] A "specific person" refers to an individual whose words and actions, code of conduct, life philosophy, past books, etc. are the subject of study.

[0933] "Speech and behavior" refers to the words and actions of a particular person.

[0934] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[0935] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[0936] "Past books" refers to books and publications written by a specific person.

[0937] "Means of learning" refers to methods and techniques for collecting and analyzing a specific person's words and actions, code of conduct, philosophy of life, past books, etc. as data.

[0938] "Question / Inquiry" refers to any inquiry or request for advice that a User enters into the System.

[0939] "Means for generating video / audio" refers to technology or methods for producing video and audio that sounds as if a specific trained person is actually speaking.

[0940] "Means of providing" refers to the methods and technologies for delivering the generated video and audio to users internally or externally.

[0941] "Brick and mortar store" refers to a commercial establishment that exists in a physical location.

[0942] A "virtual store clerk" refers to a virtual store clerk created to serve customers in a physical store.

[0943] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and run applications.

[0944] A system for implementing this invention includes means for learning the words and actions, code of conduct, philosophy of life, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person learned is speaking in response to input questions, inquiries, etc., means for providing the generated video / audio internally or externally, means for generating video / audio of a virtual store clerk answering customer questions in a physical store, and means for providing the generated video / audio to a smartphone.

[0945] Hardware and Software Used

[0946] Hardware: Smartphones, servers

[0947] Software: Python, OpenAI API, OpenCV, pyttsx3

[0948] Data processing and calculation

[0949] 1. The server collects data such as a specific person's words and actions, guidelines, life philosophy, past books, etc., and trains the system using natural language processing technology.

[0950] 2. The user uses their smartphone to enter their question or inquiry.

[0951] 3. The server uses the OpenAI API to generate answers to the questions entered, as if the specific person were actually answering them.

[0952] 4. The server converts the generated answer into an audio file using pyttsx3.

[0953] 5. The server uses OpenCV to combine the audio file with the pre-recorded video of the store clerk to generate the final video.

[0954] 6. The server provides the generated video / audio to the smartphone.

[0955] Examples

[0956] For example, when a user enters a question such as "What are the features of this product?" at a physical store, the server performs the following processing.

[0957] 1. The server uses the OpenAI API to generate an answer to the question, "What are the features of this product?"

[0958] 2. The generated answer will be, "This product uses the latest technology and is very high performance."

[0959] 3. The server converts this response into an audio file using pyttsx3.

[0960] 4. The server uses OpenCV to combine this audio file with the pre-recorded video of the store clerk to generate the final video.

[0961] 5. Users can view and hear the virtual store clerk on their smartphones.

[0962] Examples of prompt statements

[0963] Q: What are the features of this product?

[0964] A: This product uses the latest technology and is very powerful.

[0965] In this way, customers can use their smartphones to get quick and accurate answers from virtual store clerks in physical stores.

[0966] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0967] Step 1:

[0968] The user inputs a question or inquiry using a smartphone. The input question is sent to the server as text data.

[0969] Step 2:

[0970] The server sends the received question to the OpenAI API and uses a generative AI model to generate an answer to the question. The input is the user's question text and the output is the generated answer text.

[0971] Step 3:

[0972] The server converts the generated response text into an audio file using pyttsx3. The input is the response text, and the output is the audio file (e.g., response.mp3).

[0973] Step 4:

[0974] The server uses OpenCV to combine the pre-recorded video of the store attendant with the audio file to generate the final video. The input is the pre-recorded video file and the generated audio file, and the output is the final video file (e.g., response_video.mp4).

[0975] Step 5:

[0976] The server provides the generated video / audio to a smartphone. The input is the final video file, and the output is the video / audio played on the smartphone.

[0977] Step 6:

[0978] The user sees the video and audio of the virtual store assistant on his / her smartphone. The input is the video / audio file provided by the server, and the output is the user's visual and auditory reception of the information.

[0979] Example 3

[0980] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0981] In conventional systems, the means to generate the knowledge and experience of a specific person as video and audio and share it both inside and outside the company were limited. It was also difficult to efficiently store the generated video and audio and provide it at the appropriate time. This made it impossible to make the most of the knowledge and experience of a specific person.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0983] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question / consultation, etc., a means for inputting a prompt sentence into the generative AI model, a means for the generative AI model to generate video / audio, a means for saving the generated video / audio, a means for providing the generated video / audio internally or externally, a means for providing the generated video / audio to company members, and a means for publishing the generated video / audio on a website. This makes it possible to efficiently generate the knowledge and experience of a specific person as video and audio and share it both inside and outside the company.

[0984] A "specific person" refers to an individual with specific knowledge or experience.

[0985] "Speech and behavior" refers to the words and actions of a particular person.

[0986] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[0987] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[0988] "Past books" refers to books and documents written by a specific person.

[0989] "Means of learning" refers to a method for incorporating a particular person's words and actions, guidelines for behavior, life philosophy, past books, etc. into the system.

[0990] "Questions and inquiries" refers to inquiries or requests for advice that users input into the system.

[0991] "Means for generating video / audio" refers to a method for creating video and audio that sounds like a specific person is speaking based on an input question or inquiry.

[0992] A "generative AI model" refers to a model that uses artificial intelligence to generate images and sounds.

[0993] A "prompt sentence" refers to an input sentence that gives instructions to a generative AI model.

[0994] "Means of storage" refers to the method of storing the generated video and audio as data.

[0995] "Means of providing" refers to the method of distributing the generated video and audio internally or externally.

[0996] "Means of providing to internal members" refers to the method of sharing the generated video and audio with internal stakeholders.

[0997] "Means of publishing on a website" refers to a method of widely publishing the generated video and audio on the Internet.

[0998] This invention relates to a system that generates a specific person's knowledge and experience as video and audio and shares them both inside and outside the company. This system includes a means for learning the specific person's words and actions, guidelines for action, life philosophy, past books, etc., a means for generating video / audio that sounds like the specific person is speaking in response to an input question or consultation, a means for inputting a prompt sentence into a generative AI model, a means for the generative AI model to generate video / audio, a means for saving the generated video / audio, a means for providing the generated video / audio internally or externally, a means for providing the generated video / audio to company members, and a means for publishing the generated video / audio on a website.

[0999] The user inputs a prompt for the generative AI model using a dedicated application or web interface. For example, the user might input, "Please create a promotional video for our new product." The device then sends the input prompt to the server using the HTTPS protocol.

[1000] The server receives the prompt text sent from the terminal and stores it in a temporary database. The server then inputs the stored prompt text into the generative AI model and starts the generation process. Examples of generative AI models include OpenAI's GPT-4 and DALL-E. The generative AI model generates video and audio based on the prompt text. This generation process can take anywhere from a few seconds to a few minutes.

[1001] The generated video and audio are stored in the server's storage. For example, cloud storage such as Amazon S3 or Google Cloud Storage is used as the storage location. The server uploads the saved video and audio to an internal shared folder to provide it to company members, or makes it available on an internal web portal.

[1002] Furthermore, the server publishes the generated video and audio on a website, using a website management system (e.g., WordPress or Joomla) to publish the content.

[1003] As a concrete example, consider the case where a user inputs a prompt saying, "I want you to generate an interview video about a specific project." When this prompt is sent to the server, the server inputs this prompt into the generative AI model. The generative AI model generates an interview video about the project and returns it to the server. The server uploads the generated video to a shared folder within the company so that members within the company can access it. The server can also publish the video on an external website to share it widely.

[1004] Examples of prompts include:

[1005] "I want you to generate interview footage for a specific project."

[1006] "I want you to create a video to introduce our new product."

[1007] "I want you to generate a commentary audio about a specific technology."

[1008] In this way, the user can utilize the generative AI model to generate video and audio according to specific needs and share them inside and outside the company. The flow of the specification process in the third embodiment will be described with reference to FIG.

[1009] Step 1:

[1010] The user inputs a prompt for the generative AI model using a dedicated application or web interface. For example, the user might input, "Please create a promotional video for our new product." The input prompt is temporarily stored in the device's memory.

[1011] Step 2:

[1012] The terminal sends the input prompt text to the server using the HTTPS protocol. The prompt text is encrypted before being sent. The input is the prompt text, and the output is a notification of completion of transmission to the server.

[1013] Step 3:

[1014] The server receives the prompt text sent from the terminal and stores it in a temporary database. The input is the prompt text, and the output is a notification that the text has been saved to the database.

[1015] Step 4:

[1016] The server inputs the saved prompt sentence into the generative AI model and starts the generation process. The input is the prompt sentence, and the output is a notification to the generative AI model that the input has been completed.

[1017] Step 5:

[1018] A generative AI model generates video and audio based on a prompt. This generation process can take anywhere from a few seconds to a few minutes. The input is the prompt, and the output is the generated video and audio.

[1019] Step 6:

[1020] The server receives the video and audio returned from the generative AI model and stores them in the server's storage. The storage location can be cloud storage such as Amazon S3 or Google Cloud Storage. The input is the generated video and audio, and the output is a notification that the storage has been completed.

[1021] Step 7:

[1022] The server uploads the stored video and audio to a shared folder within the company to provide them to company members, or publishes them on a web portal for internal use only. The input is the stored video and audio, and the output is a notification to company members that the video and audio have been provided.

[1023] Step 8:

[1024] The server publishes the generated video and audio on a website. At this time, the content is published using a website management system (e.g., WordPress or Joomla). The input is the saved video and audio, and the output is a notification on the website that publication has been completed.

[1025] (Application example 3)

[1026] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1027] In modern society, there is a demand to widely share the knowledge and experience of specific experts and experienced people. However, there is a lack of means to efficiently collect this knowledge and experience, generate it as video or audio, and provide it to in-house members and general users. This makes it difficult to effectively communicate specialized knowledge and experience.

[1028] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the learned specific person is speaking in response to an input question / consultation, etc., a means for providing the generated video / audio internally or externally, a means for collecting knowledge and experience of specific experts and experienced people in text format, a means for creating a prompt sentence to be input to the generation AI model based on the collected text data, a means for inputting the prompt sentence to the generation AI model and generating video and audio, and a means for publishing the generated video and audio to in-house members or on a website. This makes it possible to efficiently collect the knowledge and experience of specific experts and experienced people, generate it as video and audio, and share it widely.

[1029] A "specific person" refers to a specific expert or person with experience whose knowledge or experience is important.

[1030] "Speech and behavior" refers to the words and actions of a particular person.

[1031] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[1032] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[1033] "Past books" refers to books and documents written in the past by a particular person.

[1034] "Means of learning" refers to methods or techniques for making the system learn a particular person's words and actions, code of conduct, philosophy of life, past books, etc.

[1035] "Input questions and inquiries" refers to questions and inquiries that users make to the system.

[1036] "Means for generating video / audio" refers to methods or technologies for generating video or audio that sounds like a specific trained person is speaking.

[1037] "Means of providing" refers to the methods and technologies for providing the generated video and audio internally or externally.

[1038] "Means of collecting knowledge and experience in text format" refers to methods and technologies for collecting the knowledge and experience of specific experts or experienced people as text data.

[1039] A "generative AI model" refers to an artificial intelligence model that generates video and audio based on an input prompt.

[1040] A "prompt" refers to an instruction or question to be input into a generative AI model.

[1041] "Internal members" refers to members within a particular organization or company.

[1042] "Website" refers to a web page that provides publicly available information on the Internet.

[1043] A system for implementing this invention includes means for learning the words and actions, code of conduct, life philosophy, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person who has learned is speaking in response to input questions, inquiries, etc., means for providing the generated video / audio internally or externally, means for collecting the knowledge and experience of specific experts or experienced people in text form, means for creating prompt text to be input into a generative AI model based on the collected text data, means for inputting the prompt text into the generative AI model and generating video and audio, and means for publishing the generated video and audio to internal members or on a website.

[1044] Hardware and Software Used

[1045] Hardware: Servers, smartphones, head-mounted displays

[1046] Software: Generative AI models (e.g. OpenAI GPT-4), video generation software (e.g. Adobe Premiere Pro), audio generation software (e.g. Audacity)

[1047] Data processing and calculation

[1048] 1. Data collection: The server collects the knowledge and experience of specific experts or experienced people in text format. For example, a doctor's surgical experience is collected as text data.

[1049] 2. Prompt generation: The server creates a prompt sentence to be input to the generative AI model based on the collected text data. For example, it generates a prompt sentence such as "Please explain the procedure of this surgery in detail."

[1050] 3. Video / audio generation: The server inputs the prompt text into the generative AI model to generate video and audio that sounds like a specific expert is speaking.

[1051] 4. Data provision: The server makes the generated video and audio available to company members and on a website, making it possible to share knowledge and experience widely.

[1052] Examples

[1053] For example, a doctor's surgical experience is collected and a prompt sentence that describes the surgical procedure in detail is generated. By inputting the prompt sentence "Please describe the procedure of this surgery in detail. Specifically, please include each step from preparation to completion of the surgery" into the generative AI model, video and audio of the surgical procedure are generated. The generated video and audio are published on a website for medical professionals and shared widely.

[1054] In this way, it is possible to efficiently collect the knowledge and experience of specific experts and experienced people, generate it as video and audio, and share it widely.

[1055] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1056] Step 1:

[1057] The server collects the knowledge and experience of specific experts and experienced people in text format. Specifically, it collects interviews with experts, past literature, lecture contents, etc. as text data. The input is information about the expert's knowledge and experience, and the output is text data.

[1058] Step 2:

[1059] The server creates prompt sentences to be input to the generative AI model based on the collected text data. Specifically, it analyzes the collected text data, extracts important keywords and phrases, and generates prompt sentences. The input is the collected text data, and the output is the generated prompt sentence.

[1060] Step 3:

[1061] The server inputs a prompt sentence into the generative AI model to generate video and audio. Specifically, the server inputs a prompt sentence into the generative AI model and executes a process in which the model generates video and audio. The input is the generated prompt sentence, and the output is the generated video and audio.

[1062] Step 4:

[1063] The server makes the generated video and audio available to company members or on a website. Specifically, it converts the generated video and audio into an appropriate format and uploads it to an internal shared system or website. The input is the generated video and audio, and the output is the published video and audio.

[1064] Step 5:

[1065] A user views publicly available video and audio and learns from the knowledge and experience of a specific expert or person with experience. Specifically, a user accesses a website and plays the publicly available video and audio. The input is the publicly available video and audio, and the output is the user's learning outcome.

[1066] In addition, an emotion engine that estimates the emotion of the user may be further combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[1067] "Example 1"

[1068] One embodiment of the present invention is a system that incorporates an emotion engine. This system automatically collects public information about a specific person and learns the person's words and actions, guidelines for action, and philosophy of life. Furthermore, the emotion engine that recognizes the user's emotions is used to respond according to the user's emotions. Specifically, when the user is feeling happy, a video / audio is generated in which a specific person shares the joy, and when the user is feeling sad, a video / audio is generated in which a specific person comforts the user.

[1069] "Example 2"

[1070] The generated video / audio can be provided internally or externally. Specifically, for internal members, it can be provided via email or the internal network, and published on a website as a streaming service or available for download. At this time, it is also possible to change the content provided according to the user's emotions. For example, when a user is feeling happy, content is provided in which a specific person shares the joy, and when the user is feeling sad, content is provided in which a specific person comforts the user.

[1071] "Example 3"

[1072] Furthermore, the emotion engine can recognize changes in the user's emotions in real time and generate video / audio in response to those changes. For example, if a user starts out feeling happy but then changes to sadness, the emotion engine will recognize that change and generate video / audio of a specific person comforting them. This allows for an appropriate response to the user's emotions.

[1073] The process flow of each embodiment will be described below.

[1074] "Example 1"

[1075] Step 1: Automatically collect publicly available information about a specific person.

[1076] Step 2: Based on the information you have collected, learn about the person's words and actions, code of conduct, and life philosophy.

[1077] Step 3: Using an emotion engine that recognizes the user's emotions, respond according to the user's emotions.

[1078] Step 4: Generate video / audio that sounds like a specific person is speaking depending on the user's emotions.

[1079] "Example 2"

[1080] Step 1: Provide the generated video / audio internally or externally.

[1081] Step 2: Change what you offer based on the user's emotions.

[1082] Step 3: Depending on the user's emotions, provide content that sounds like a specific person is speaking.

[1083] "Example 3"

[1084] Step 1: Recognize changes in user emotions in real time.

[1085] Step 2: Generate video / audio according to changes in emotions.

[1086] Step 3: Respond appropriately to the user's emotions.

[1087] Example 1

[1088] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1089] Conventional systems have had difficulty not only learning a specific person's words and actions, code of conduct, and life philosophy, but also responding according to the user's emotions. In addition, there was a lack of technology to recognize the user's emotions and generate images and sounds accordingly. This made it difficult to provide a more personalized experience for users.

[1090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1091] In this invention, the server includes a means for learning the words and actions, guidelines for action, life philosophy, past books, etc. of a specific person, a means for automatically collecting public information of the specific person, a means for recognizing the user's emotions, a means for generating video / audio that sounds like the specific person is speaking according to the user's emotions, and a means for providing the generated video / audio internally or externally. This makes it possible to provide personalized video and audio according to the user's emotions.

[1092] A "specific person" refers to a particular individual or character, whose words and actions, code of conduct, and life philosophy are the subject of study.

[1093] "Behavior" refers to the words and actions of a particular person, and represents that person's characteristics and personality.

[1094] A "code of conduct" refers to the principles or policies of behavior that a particular person follows, which serve as the standard for that person's decision-making and actions.

[1095] "Philosophy of life" refers to a particular person's outlook on life and values, which form the basis of that person's way of life and way of thinking.

[1096] "Past books" refer to books written by a particular person or books about that person, and serve as sources of information for learning.

[1097] "Public information" refers to information that is available to the public, such as Internet articles, books, videos, and audio.

[1098] "Means for recognizing emotions" refers to technology or devices for analyzing a user's voice and facial expression data and identifying the emotion.

[1099] "Means for generating video / audio" refers to technology or equipment for creating video or audio that sounds like a specific person is speaking.

[1100] "Means of providing" refers to the technology and devices for providing the generated video and audio to the user.

[1101] "Internal or external" refers to devices or networks within the system, or the external environment, such as the Internet.

[1102] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1103] The present invention is a system that learns the words and actions of a specific person, guidelines for action, life philosophy, past books, etc., and generates and provides images and sounds according to the user's emotions. A specific embodiment of this system will be described below.

[1104] 1. Program Generation

[1105] The server automatically collects public information about a specific person and generates a program that learns from that information. The program targets articles, books, videos, audio, etc. on the Internet. The software used includes BeautifulSoup and Scrapy as web crawlers, MySQL and MongoDB as databases, and TensorFlow and PyTorch as deep learning frameworks.

[1106] 2. Program Processing Explanation

[1107] The server uses the generated program to collect public information about a specific person. The collected information is stored in a database. The server then uses deep learning technology to analyze the collected information and learn the specific person's words and actions, code of conduct, and life philosophy.

[1108] In addition, the server runs a system that combines an emotion engine that analyzes the user's voice and facial expression data to recognize the user's emotions. The emotion engine uses software such as OpenCV and Emotion API.

[1109] When a user accesses the system, the terminal transmits the user's emotions to the emotion engine. The emotion engine analyzes the user's emotions and transmits the results to the server. Based on the analysis results, the server generates images and audio for a specific person to respond according to the user's emotions. A generative AI model is used to generate images and audio based on the prompt text.

[1110] 3. Specific Examples

[1111] As a concrete example, consider the case where a specific person is a famous author. The server collects interview articles and lecture videos of the author, and uses deep learning to learn the author's life philosophy. When a user accesses the system and the emotion engine recognizes the user's joy, the server generates video and audio that show the author sharing his or her joy, and sends them to the terminal.

[1112] Example prompt:

[1113] "Collect interviews and lecture videos of famous authors and learn their life philosophies. If the user feels joy, generate video or audio of the author sharing his or her joy."

[1114] In this way, the system operates in cooperation with the server, terminals, and users.

[1115] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1116] Step 1: Gather public information

[1117] The server collects public information about a particular person from the Internet. As input, it uses the name of the particular person and related keywords. Specifically, it uses a web crawler (e.g., BeautifulSoup or Scrapy) to collect information such as articles, books, videos, and audio. As output, it obtains the collected information in the form of text, images, audio, etc.

[1118] Step 2: Save your data

[1119] The server stores the collected information in a database. As input, it uses the information collected in step 1. Specifically, it executes SQL queries to a database such as MySQL or MongoDB to store text data and metadata. As output, it gets the information stored in the database.

[1120] Step 3: Analyze and train data

[1121] The server analyzes the stored data and learns the words and actions, behavioral guidelines, and life philosophy of a particular person. It uses the information stored in the database as input. Specifically, it uses TensorFlow and PyTorch to build a deep learning model and trains the data. The output is a trained model.

[1122] Step 4: Recognize emotions

[1123] The device collects the user's voice and facial expression data and sends it to the emotion engine. As input, the device uses the user's voice and facial expression data. Specifically, the device captures the data using a camera and microphone, and analyzes the emotions using OpenCV and the Emotion API. As output, the analyzed emotion data is obtained.

[1124] Step 5: Generate correspondence

[1125] Based on the analysis results received from the emotion engine, the server generates video and audio for a specific person to respond according to the user's emotions. Analyzed emotion data and a trained model are used as input. Specifically, a prompt sentence is input into the generative AI model to generate video and audio. The generated video and audio data is obtained as output.

[1126] Step 6: Providing a response

[1127] The server transmits the generated video and audio to the terminal. The generated video and audio data is used as input. Specifically, the data is transmitted to the terminal via a network, and the terminal provides the received video and audio to the user. The video and audio viewed by the user are obtained as output.

[1128] (Application example 1)

[1129] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1130] Conventional systems were able to learn the words, actions, and guidelines of a specific person and generate video and audio that sounded like that person was speaking in response to a user's questions or concerns. However, they were unable to respond appropriately according to the user's emotions, and improving the user experience was an issue. In addition, since the generation of prompt sentences using a generative AI model was not performed, more advanced responses based on information about specific people were required.

[1131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1132] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question or consultation, a means for providing the generated video / audio internally or externally, a means for generating video / audio according to the user's emotions using an emotion engine that recognizes the user's emotions, and a means for generating a prompt sentence based on information about the specific person using a generative AI model. This enables an appropriate response according to the user's emotions and realizes an advanced response based on information about the specific person.

[1133] A "specific person" is an individual whose words, actions, code of conduct, and life philosophy are to be studied based on publicly available information.

[1134] "Speech and behavior" refers to the words and actions of a particular person.

[1135] A "code of conduct" refers to the basic ideas or policies that a particular person follows when acting.

[1136] A "philosophy of life" refers to a person's basic thoughts and values ​​regarding life.

[1137] "Past books" refer to books written by a particular person or books related to that person.

[1138] "Means of learning" refers to a method or device for learning a specific person's words and actions, code of conduct, life philosophy, past books, etc. using machine learning technology.

[1139] "Questions and inquiries" refer to inquiries or requests for advice that users input into the system.

[1140] "Means for generating video / audio" refers to a method or device for generating video and audio that sounds like a specific trained person is speaking.

[1141] The "means for providing" refers to a method or device for providing the generated video and audio internally or externally.

[1142] An "emotion engine" is a technology or device that recognizes a user's emotions and responds according to those emotions.

[1143] A "generative AI model" is an artificial intelligence model that generates prompt sentences based on information about a specific person.

[1144] A "prompt sentence" is a sentence that is generated using a generative AI model based on information about a specific person.

[1145] A system for implementing this invention includes means for learning the words and actions, code of conduct, philosophy of life, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person is speaking in response to an input question, consultation, etc., means for providing the generated video / audio internally or externally, means for generating video / audio in accordance with the user's emotions using an emotion engine that recognizes the user's emotions, and means for generating prompt sentences based on information about a specific person using a generative AI model.

[1146] Explanation of system program processing

[1147] The server retrieves data from the Internet to collect public information about a specific person. The collected data is trained using a deep learning model (e.g., TensorFlow or PyTorch). The trained model is used to understand the specific person's behavior, course of action, and life philosophy.

[1148] When a user inputs a question or inquiry into the system, the server uses an emotion engine to recognize the user's emotions. Based on the recognized emotions, a prompt sentence is generated using a generative AI model (e.g., GPT-3). This prompt sentence is based on information about a specific person and is intended to generate an appropriate response according to the user's emotions.

[1149] Based on the generated prompt, the server generates video and audio that sounds like a specific person is speaking. The generated video and audio are provided to the user via their smartphone or a website.

[1150] Examples

[1151] For example, consider the case of collecting information on famous authors and generating content that would comfort a user when the author is sad. The server collects public information on authors and learns that information using a deep learning model. When a user enters "I've recently failed at work and I'm feeling depressed," the emotion engine recognizes the user's sadness. The generative AI model generates a prompt sentence like the following:

[1152] Collect public information about famous authors and generate content that will comfort your users when they are sad.

[1153] Based on this prompt, the server generates images and sounds that are comforting to the user, allowing the user to receive an appropriate response from a specific person and obtain emotional support.

[1154] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1155] Step 1:

[1156] The server collects public information about a particular person.

[1157] Input: Name of a specific person

[1158] Specific operation: The server uses an API on the Internet to retrieve public information such as articles, books, videos, and audio related to a particular person.

[1159] Output: A dataset of collected public information

[1160] Step 2:

[1161] The server trains a deep learning model based on the collected public information.

[1162] Input: A dataset of collected public information

[1163] How it works: The server uses deep learning frameworks such as TensorFlow and PyTorch to train a model that learns a particular person's words, actions, code of conduct, and life philosophy.

[1164] Output: A trained deep learning model

[1165] Step 3:

[1166] A user inputs a question or inquiry into the system.

[1167] Input: The text of the user's question or request

[1168] Specific actions: The user inputs a question or inquiry through a smartphone or website interface.

[1169] Output: Text data of user questions and inquiries

[1170] Step 4:

[1171] The server recognizes the user's emotions using an emotion engine.

[1172] Input: Text data of user questions and inquiries

[1173] Specific operation: The server uses the EmotionEngine to analyze emotions from the user's text data.

[1174] Output: User emotion data (e.g. happy, sad, angry, etc.)

[1175] Step 5:

[1176] The server generates a prompt sentence using a generative AI model.

[1177] Input: User emotion data, information about specific people

[1178] Specific operation: The server uses a generative AI model such as GPT-3 to generate prompt sentences that correspond to the user's emotions based on information about a specific person.

[1179] Output: Generated prompt statement

[1180] Step 6:

[1181] The server generates video and audio that sounds like a specific person is speaking based on the prompt text.

[1182] Input: Generated prompt sentence, trained deep learning model

[1183] Specific operation: The server uses the generated prompt text to run an algorithm to generate video and audio that sounds like a specific person is speaking.

[1184] Output: The generated video and audio data

[1185] Step 7:

[1186] The server provides the generated video and audio to the user.

[1187] Input: Generated video and audio data

[1188] Specific operation: The server distributes the generated video and audio to the user via a smartphone or website.

[1189] Output: Video and audio data that can be viewed by the user

[1190] Example 2

[1191] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[1192] Conventional systems could generate video / audio that sounded as if a specific person was answering a user's questions or concerns, but they could not change the content according to the user's emotions. This made it difficult to provide an appropriate answer that was in line with the user's emotions. In addition, there were limited means to effectively provide the generated video / audio.

[1193] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1194] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question or inquiry, a means for analyzing the user's emotions, a means for changing the content of the generated video / audio based on the analyzed emotions, and a means for providing the generated video / audio internally or externally. This makes it possible to provide an appropriate answer according to the user's emotions and to effectively provide the generated video / audio.

[1195] A "specific person" is an individual that the system learns from, and is the subject of video / audio generation based on that individual's words and actions, code of conduct, philosophy of life, past books, etc.

[1196] "Behavior" refers to what a particular person says or does, and is part of the data that the system learns.

[1197] "Code of conduct" refers to the principles or policies of behavior that a particular person should follow, and is part of the data that the system learns.

[1198] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life, and is part of the data that the system learns.

[1199] "Past books" refers to books written by or about a particular person, and are part of the data the system learns from.

[1200] "Means of learning" refers to techniques and methods for importing a particular person's words and actions, code of conduct, philosophy of life, past books, etc. into the system and analyzing them.

[1201] "Input questions and inquiries" refers to questions and inquiries submitted by users to the system.

[1202] "Means for generating video / audio" refers to technology or methods for creating video and audio that sounds as if a specific trained person is answering an input question or inquiry.

[1203] "Means for analyzing user emotions" refers to techniques and methods for reading and analyzing emotions from user input data.

[1204] "Means for changing the content of generated video / audio" refers to techniques or methods for adjusting the content of generated video and audio based on the analyzed emotions of the user.

[1205] "Means of delivery" refers to the technology or methods used to deliver the generated video / audio to users or other recipients.

[1206] "Providing internally or externally" refers to distributing the generated video / audio to company members or external users.

[1207] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1208] The present invention is a system that generates video / audio that sounds like a specific person is speaking and changes the content according to the user's emotions. A specific embodiment of this system is described below.

[1209] System Configuration

[1210] This system consists of three main elements: a server, a terminal, and a user. The server runs a program that combines natural language processing technology and audio / video generation technology. The terminal is a device through which users input their questions or inquiries, and can include PCs, smartphones, and tablets. The user is an individual who uses the system to ask questions or make inquiries.

[1211] Hardware and Software Used

[1212] The server is a computer equipped with a high-performance processor and a large amount of memory, and uses the following software:

[1213] Natural language processing models: Google's BERT, OpenAI's GPT-4

[1214] Speech generation technology: Amazon Polly, Google Text-to-Speech

[1215] Video generation technology: Deepfake technology, GAN (Generative Adversarial Network)

[1216] Sentiment analysis technology: Microsoft Azure's Emotion API, Google Cloud's Natural Language API

[1217] System Operation

[1218] The user uses a terminal to input a question or inquiry into the system's interface, for example through a form on a web browser or a dedicated application. The input data is then sent to the server.

[1219] The server analyzes the received text data of the question or inquiry using a natural language processing model (e.g., Google's BERT or OpenAI's GPT-4). This analysis allows the intent and content of the question or inquiry to be understood.

[1220] Next, the server analyzes the user's input data to determine emotions. For emotion analysis, it uses Microsoft Azure's Emotion API and Google Cloud's Natural Language API. Based on the analysis results, it identifies the user's emotions, such as whether they are happy, sad, or angry.

[1221] The server generates video / audio that sounds like a specific person is answering based on the analyzed question or consultation content and the user's emotions. Amazon Polly and Google Text-to-Speech are used for voice generation, and Deepfake technology and GAN are used for video generation. For example, if a user enters "I would like some advice on a new project" and emotion analysis determines that the user is "excited," the server generates video / audio that sounds like a specific business consultant is sharing his / her excitement.

[1222] The generated video / audio is provided to users by the server. It is provided to company members via email or the company network, and is made available on websites via streaming services such as YouTube and Vimeo, or by providing a direct download link. For example, the link to the generated video / audio is sent by email or uploaded to an internal portal site.

[1223] Examples and prompts

[1224] As a concrete example, consider a case where a user asks, "I would like some advice on a new project." The server analyzes the question and generates video and audio that sounds as if a specific person (e.g., a well-known business consultant) is answering the question.

[1225] Example prompt:

[1226] "I'd like some advice on a new project. Can you generate video / audio of a specific person answering?"

[1227] In this way, users can have the experience of feeling as if a specific person is actually answering their questions or inquiries.

[1228] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1229] Step 1:

[1230] The user inputs a question or inquiry.

[1231] A user uses a terminal (PC, smartphone, tablet, etc.) to input a question or request for advice into the system's interface. For example, the user might enter "I'd like some advice on a new project" through a form on a web browser or a dedicated application, and then click the send button. The input data is sent to the server in text format.

[1232] Step 2:

[1233] The server receives the input data and performs natural language processing.

[1234] The server receives text data of questions or inquiries sent by users. The received data is analyzed using a natural language processing model (e.g., Google's BERT or OpenAI's GPT-4). Through this analysis, the intent and content of the question or inquiry are understood. The input is the user's text data, and the output is data containing the analyzed intent and content.

[1235] Step 3:

[1236] The server analyzes the user's emotions.

[1237] The server analyzes emotions from the user's input data. For emotion analysis, it uses Microsoft Azure's Emotion API and Google Cloud's Natural Language API. Based on the analysis results, it identifies the user's emotions, such as whether they are happy, sad, or angry. The input is the user's text data, and the output is the analyzed emotion data.

[1238] Step 4:

[1239] The server generates video / audio of a specific person.

[1240] The server generates video / audio that sounds as if a specific person is answering based on the analyzed question or consultation content and the user's emotions. Amazon Polly or Google Text-to-Speech is used for audio generation, and Deepfake technology or GAN (Generative Adversarial Network) is used for video generation. For example, if a user enters "I would like some advice about a new project" and emotion analysis determines that the user is "excited," video / audio is generated in which a specific business consultant shares their excitement. The input is the analyzed intent and emotion data, and the output is the generated video / audio data.

[1241] Step 5:

[1242] The server provides the generated video / audio.

[1243] The server provides the generated video / audio to users. It provides it to internal members via email or the internal network, and makes it available on websites via streaming services such as YouTube or Vimeo, or by providing a direct download link. For example, the link to the generated video / audio is sent by email or uploaded to an internal portal site. The input is the generated video / audio data, and the output is the link or file provided to the user.

[1244] (Application example 2)

[1245] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1246] Conventional Q&A systems have difficulty in providing personalized answers according to the user's emotions, limiting the user experience. In addition, there is a lack of technology to generate realistic images and audio that make it seem as if a specific person is answering. This makes it difficult for users to obtain the specific, emotionally sensitive answers they desire.

[1247] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1248] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person is speaking in response to an input question / consultation, a means for recognizing the user's emotions, a means for adjusting the content and tone of the video / audio generated according to the user's emotions, and a means for providing the generated video / audio internally or externally. This makes it possible to provide a personalized answer according to the user's emotions and generate realistic video and audio that sounds like a specific person is answering.

[1249] A "specific person" is an individual, such as a celebrity or an expert, who is set as a respondent when a user asks a question or makes a request for advice.

[1250] "Words and actions" are records of words and actions that a particular person has made in the past.

[1251] A "code of conduct" is a principle or policy of behavior that a particular person should follow.

[1252] A "philosophy of life" is a person's way of thinking and values ​​regarding life.

[1253] "Past books" are books or publications written by a particular person.

[1254] "Means of learning" refers to technology for collecting and analyzing data such as a specific person's words and actions, code of conduct, philosophy of life, past books, etc.

[1255] "Questions and inquiries" are questions or concerns that users input into the system.

[1256] "Means for generating video / audio" refers to technology that creates video and audio that sounds as if a specific person is actually speaking.

[1257] "Means of recognizing emotions" refers to technology that analyzes emotions from a user's facial expressions and voice.

[1258] "Means for adjusting content and tone" refers to technology that changes the content and tone of generated video and audio in response to the user's emotions.

[1259] "Means of providing" refers to the technology used to deliver the generated video and audio to the user.

[1260] "Internal or external" refers to users within the system or external websites.

[1261] A system for implementing this invention includes means for learning the words and actions, code of conduct, philosophy of life, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person learned is speaking in response to an input question / consultation, etc., means for recognizing a user's emotions, means for adjusting the content and tone of the generated video / audio in response to the user's emotions, and means for providing the generated video / audio internally or externally.

[1262] System Configuration

[1263] The server uses the following hardware and software:

[1264] Hardware: PC with high-performance CPU and GPU, web camera

[1265] Software: Python, OpenAI API, OpenCV, EmotionRecognizer (emotion recognition library)

[1266] Data processing and calculation

[1267] 1. Entering a question: The user enters a question or inquiry through the terminal.

[1268] 2. Emotion Recognition: The server captures the user's facial expressions using a webcam and recognizes the emotions using the EmotionRecognizer library.

[1269] 3. Answer Generation: The server uses the OpenAI API to generate an answer based on the question and the user's sentiment. Specifically, it generates a prompt sentence like the following:

[1270] Example prompt:

[1271] User Question: I've been having trouble at work lately. What should I do?

[1272] User Emotion: Sadness

[1273] Generate content for a specific person to answer.

[1274] 4. Video Generation: The server uses OpenCV to generate video frames containing the generated answer text and saves them as video files.

[1275] 5. Provision: The server provides the generated video and audio to the user in streaming or download format.

[1276] Examples

[1277] If a user inputs the question "I'm worried that things haven't been going well at work lately. What should I do?" and the emotion recognition system recognizes the user's emotion as "sadness," the server will generate the following prompt sentence:

[1278] User Question: I've been having trouble at work lately. What should I do?

[1279] User Emotion: Sadness

[1280] Generate content for a specific person to answer.

[1281] Based on this prompt, the OpenAI API is used to generate a response that sounds like a specific person is answering, and a video containing that response is generated. The generated video is provided with content and tone that is in tune with the user's emotions.

[1282] The flow of the identification process in the application example 2 will be described with reference to FIG.

[1283] Step 1:

[1284] The user inputs questions or inquiries through the terminal.

[1285] Input: Text data of user's question or inquiry

[1286] Output: The text data of the question or inquiry is sent to the server.

[1287] Specific operation: The user enters a question or inquiry into the input form on the terminal and presses the send button. The terminal sends the entered text data to the server.

[1288] Step 2:

[1289] The server captures the user's facial expressions using a webcam and recognizes emotions using the EmotionRecognizer library.

[1290] Input: Video data of the user's facial expression

[1291] Output: User emotion data (e.g. happy, sad, angry, etc.)

[1292] Specific operation: The server starts the webcam and captures the user's facial expressions in real time. The captured video data is input to the EmotionRecognizer library and the emotions are analyzed. Emotion data is obtained as the analysis result.

[1293] Step 3:

[1294] The server uses the OpenAI API to generate answers based on the question and the user's sentiment.

[1295] Input: Text data of questions and inquiries, user emotion data

[1296] Output: Answer text data

[1297] Specific operation: The server generates a prompt sentence by combining the text data of the question or inquiry with the user's emotion data. The generated prompt sentence is sent to the OpenAI API, and an answer is generated as if a specific person were answering. The text data of the generated answer is obtained.

[1298] Step 4:

[1299] The server uses OpenCV to generate video frames containing the generated answer text and saves them as video files.

[1300] Input: Text data of the answer

[1301] Output: Video file (movie file)

[1302] What happens: The server uses OpenCV to generate video frames that show the answer text, and then saves the frames in sequence as a video file.

[1303] Step 5:

[1304] The video and audio generated by the server are provided to the user in streaming or download format.

[1305] Input: Video file (movie file)

[1306] Output: Video data that can be viewed by the user

[1307] Specific operation: The server uploads the generated video file to the streaming server and generates a URL that users can access. Users can watch the video through the URL. Also, the server provides the video file in a download format if necessary.

[1308] Example 3

[1309] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1310] Conventional systems could generate video and audio by utilizing the knowledge and experience of a specific person, but it was difficult to respond appropriately to the user's emotions. In addition, there was a problem that the quality of the user experience was reduced because the system could not respond to changes in the user's emotions in real time. Furthermore, there were limited means to effectively provide the generated video and audio, so knowledge and experience were not shared sufficiently.

[1311] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1312] In this invention, the server includes a means for learning the words and actions, guidelines for action, philosophy of life, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question / consultation, etc., a means for collecting user emotion data, a means for analyzing the collected emotion data, a means for generating a prompt sentence based on the analysis result, a means for generating video / audio based on the generated prompt sentence, and a means for providing the generated video / audio internally or externally. This makes it possible to generate and provide appropriate video and audio according to the user's emotions, and to effectively utilize and widely share the knowledge and experience of a specific person.

[1313] A "specific person" refers to an individual with specific knowledge or experience.

[1314] "Behavior" refers to the words and actions of a particular person.

[1315] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[1316] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[1317] "Past books" refers to books and publications written by a specific person.

[1318] "Means of learning" refers to a method of inputting a specific person's words and actions, code of conduct, philosophy of life, past books, etc. into the system and having it learn from them.

[1319] "Questions and inquiries" refers to inquiries or requests for advice that users input into the system.

[1320] "Means for generating video / audio" refers to a method for producing video or audio based on learned content.

[1321] "Emotion data" refers to information about emotions derived from a user's facial expressions, tone of voice, etc.

[1322] "Means of collection" refers to methods of obtaining emotional data using devices such as cameras and microphones.

[1323] "Means for analyzing" refers to a method for analyzing collected emotion data to recognize a user's emotional state.

[1324] A "prompt sentence" refers to an instruction sentence to be input into a generative AI model.

[1325] "Means of generation" refers to the method of producing images and sounds based on the prompt text.

[1326] "Means of providing" refers to the method of distributing the generated video and audio internally or externally.

[1327] "Internal" refers to users and members within the system.

[1328] "External" refers to public access or websites outside the system.

[1329] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1330] This invention is a system that utilizes the knowledge and experience of a specific person to generate and provide video and audio corresponding to the user's emotions. A specific embodiment of this system is described below.

[1331] System Configuration

[1332] The system is composed of three main elements: a server, a terminal, and a user. The server includes a means for learning a specific person's words and actions, guidelines for action, life philosophy, past books, etc., a means for analyzing emotion data, a means for generating prompts, a means for generating video / audio, and a means for providing the generated video / audio. The terminal includes a means for collecting user emotion data.

[1333] Hardware and Software Used

[1334] The server is a computer system equipped with a high-performance processor and large-capacity memory. Specifically, it uses NVIDIA's GPU to perform high-speed analysis of emotion data. OpenAI's GPT-4 is used as the generative AI model. The terminal is a device equipped with a camera and microphone that collects the user's facial expressions and tone of voice in real time.

[1335] Data processing and calculation

[1336] The device collects the user's facial expressions and tone of voice through a camera and microphone. The collected data is sent to a server. The server analyzes the data using an emotion engine to recognize the user's emotional state. Based on the analysis results, the server generates a prompt sentence to be input into a generative AI model. The generative AI model generates video and audio based on the prompt sentence. The generated video and audio are provided internally or externally through the server.

[1337] Examples

[1338] For example, if a user is confused during a presentation, the device collects the user's confused facial expression and trembling voice. The server uses an emotion engine to recognize that the user is confused and generates a prompt sentence such as "The user is confused, so please generate an encouraging message." Based on this prompt sentence, the generative AI model generates an encouraging message such as "It's okay, let's move on to the next question." The generated message is displayed in real time on the user's device to support the user.

[1339] Example prompt:

[1340] The user is confused, so generate an encouraging message.

[1341] In this way, a system is realized in which the server, the terminal, and the user work together to generate and provide appropriate video and audio in accordance with the user's emotions. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1342] Step 1:

[1343] The device collects the user's emotional data. Specifically, it captures the user's facial expressions with a camera and records the tone of voice with a microphone. This allows the user's facial movements and voice changes to be acquired as data in real time. The input is the user's facial expressions and voice, and the output is the collected emotional data.

[1344] Step 2:

[1345] The terminal transmits the collected emotion data to the server. Specifically, the terminal transfers the collected emotion data to the server through a network. The input is the collected emotion data, and the output is the emotion data transmitted to the server.

[1346] Step 3:

[1347] The server analyzes the emotion data. Specifically, it uses an emotion engine to analyze the data and recognize the user's emotional state. For example, if the user's eyebrows are furrowed or their voice is trembling, the emotion engine recognizes that the user is confused. The input is the emotion data sent to the server and the output is the analyzed emotional state.

[1348] Step 4:

[1349] The server generates a prompt sentence based on the analysis result. Specifically, it generates a prompt sentence such as "The user is confused, so please generate an encouraging message" based on the emotional state obtained from the emotion engine. The input is the analyzed emotional state, and the output is the generated prompt sentence.

[1350] Step 5:

[1351] The server inputs a prompt sentence into the generative AI model and generates video and audio. Specifically, the server inputs a prompt sentence into a generative AI model (for example, OpenAI's GPT-4) and generates video and audio according to the user's emotions. For example, it generates an encouraging message such as "It's okay, let's move on to the next question." The input is the generated prompt sentence, and the output is the generated video and audio.

[1352] Step 6:

[1353] The server provides the generated video and audio. Specifically, the generated video and audio are provided to company members or published on a website. For example, a generated message of encouragement is displayed on the user's terminal in real time to support the user. The input is the generated video and audio, and the output is the provided video and audio.

[1354] (Application example 3)

[1355] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1356] Conventional systems can generate video and audio to utilize the knowledge and experience of a specific person, but they have the problem of being unable to respond appropriately to changes in the user's emotions. This means that the system is unable to respond appropriately to the user's emotions, resulting in a lack of improvement in the user experience.

[1357] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a means for learning the words and actions, guidelines for action, philosophy of life, past books, etc. of a specific person, a means for generating video / audio that sounds like the learned specific person is speaking in response to an input question / consultation, etc., a means for providing the generated video / audio internally or externally, a means for recognizing changes in the user's emotions in real time, a means for generating video / audio in response to changes in emotions, and a means for playing the generated video / audio. This makes it possible to generate and provide appropriate video and audio in real time according to the user's emotions.

[1358] A "specific person" refers to an individual with specific knowledge or experience.

[1359] "Speech and behavior" refers to the words and actions of a particular person.

[1360] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[1361] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[1362] "Past books" refers to books and documents written by a specific person.

[1363] "Means of learning" refers to methods or techniques for making a system learn a particular person's words and actions, code of conduct, philosophy of life, past books, etc.

[1364] "Input questions and inquiries" refers to the content of questions and inquiries that users make to the system.

[1365] "Means for generating video / audio" refers to methods or technologies for producing video or audio based on learned content.

[1366] "Means of providing" refers to the methods and technologies for providing the generated video and audio internally or externally.

[1367] "Means for recognizing changes in emotions in real time" refers to a method or technology for detecting changes in a user's emotions in real time.

[1368] "Means for generating video / audio in response to changes in emotions" refers to methods or technologies for creating appropriate video and audio based on changes in a user's emotions.

[1369] "Video / audio playback means" refers to the methods and techniques for playing the generated video and audio to the user.

[1370] A system for implementing this invention includes means for learning the words and actions, code of conduct, philosophy of life, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person who has been learned is speaking in response to an input question or inquiry, means for providing the generated video / audio internally or externally, means for recognizing changes in a user's emotions in real time, means for generating video / audio in response to changes in emotions, and means for playing back the generated video / audio.

[1371] System Programming

[1372] The program of this system performs the following processes.

[1373] Hardware and Software

[1374] Hardware: Smartphone camera

[1375] Software: OpenCV (face detection), Keras (emotion recognition model), text_to_speech (audio generation), video_generator (video generation)

[1376] Data processing and calculation

[1377] The server captures video in real time using the smartphone camera. It detects faces using OpenCV and converts the detected face images to grayscale. It then predicts emotions from the face images using Keras emotion recognition models. Depending on the predicted emotions, it generates appropriate audio using the text_to_speech library and appropriate video using the video_generator library. The generated video and audio are then played back to the user.

[1378] Examples

[1379] For example, when a user is using the app through their smartphone camera, if the emotion recognition model detects "sadness" from the user's facial expression, the server will generate audio saying, "I see you are feeling sad. Here is something to comfort you," and play a video comforting the user.

[1380] Examples of prompt statements

[1381] "When a user is using the app through the smartphone camera, if the emotion recognition model detects a specific emotion from the user's facial expression, please create an application that generates and plays audio and video corresponding to that emotion."

[1382] In this way, it becomes possible to generate and provide appropriate video and audio in real time according to the user's emotions.

[1383] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1384] Step 1:

[1385] The server captures video in real time using the smartphone camera. The input is video data from the camera, and the output is the captured video frames. Specifically, the camera starts up and continuously captures video including the user's face.

[1386] Step 2:

[1387] The server uses OpenCV to detect faces from the captured video frames. The input is the video frame obtained in step 1, and the output is the area information (coordinates) of the detected face. The specific operation is to convert the video frame to grayscale and apply a face detection algorithm to identify the position of the face.

[1388] Step 3:

[1389] The server cuts out the detected face area and converts it into a grayscale image. The input is the face area information obtained in step 2 and the original video frame, and the output is a grayscale image of the face. The specific operation is to cut out the face area and resize it to 48x48 pixels.

[1390] Step 4:

[1391] The server uses a Keras emotion recognition model to predict emotions from grayscale facial images. The input is the grayscale facial image obtained in step 3, and the output is the predicted emotion label. The specific operation is to input the facial image into the model, calculate the probability distribution of emotions, and select the emotion label with the highest probability.

[1392] Step 5:

[1393] The server generates an appropriate speech according to the predicted emotion using the text_to_speech library. The input is the emotion label obtained in step 4, and the output is the generated speech data. The specific operation is to select a text message corresponding to the emotion label and convert the text to speech.

[1394] Step 6:

[1395] The server generates an appropriate video according to the predicted emotion using the video_generator library. The input is the emotion label obtained in step 4, and the output is the generated video data. Specifically, the server selects a video template corresponding to the emotion label and generates a video based on the template.

[1396] Step 7:

[1397] The server plays the generated video and audio to the user. The input is the audio data obtained in step 5 and the video data obtained in step 6, and the output is the video and audio played to the user. In concrete terms, the video and audio are played simultaneously using the smartphone's display and speaker.

[1398] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1399] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1400] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1401] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1402] [Third embodiment]

[1403] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1404] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1405] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1406] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1407] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1408] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1409] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1410] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1411] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1412] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1413] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1414] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1415] "Example 1"

[1416] As one embodiment of the present invention, there is a system equipped with a means for automatically collecting public information of a specific person, such as articles, books, videos, audio, etc. on the Internet. This system learns the words and actions, guidelines for action, and life philosophy of the specific person based on the collected information. Machine learning techniques such as deep learning are used for the learning.

[1417] "Example 2"

[1418] Furthermore, as one embodiment of the present invention, there is a system equipped with a means for generating video / audio that sounds as if a specific person who has been trained is speaking in response to an input question / consultation, etc. This system combines natural language processing technology and audio / video generation technology to generate video / audio that sounds as if a specific person is actually answering the question / consultation.

[1419] "Example 3"

[1420] As one embodiment of the present invention, there is a system equipped with a means for providing the generated video / audio internally or externally. This system makes it possible to utilize the knowledge and experience of a specific person by providing the generated video / audio to members of the company. In addition, by publishing the generated video / audio on a website, it becomes possible to share that knowledge and experience widely.

[1421] The process flow of each embodiment will be described below.

[1422] "Example 1"

[1423] Step 1: Automatically collect public information about a specific person, such as articles, books, videos, audio, etc. on the Internet. This collection is done using web scraping technology and APIs.

[1424] Step 2: Based on the collected information, the behavior, code of conduct, and life philosophy of a specific person are learned. This learning is done using machine learning techniques such as deep learning. Specifically, a neural network is trained to generate a model that mimics the behavioral patterns of a specific person.

[1425] "Example 2"

[1426] Step 1: Receive input from the user, such as a question or inquiry. This input can be in text or voice format.

[1427] Step 2: In response to the input question or inquiry, video / audio is generated that sounds as if a specific person who has been trained is speaking. This generation is performed by combining natural language processing technology and audio / video generation technology. Specifically, using the trained model, video / audio is generated that sounds as if a specific person is actually answering the question or inquiry.

[1428] "Example 3"

[1429] Step 1: The generated video / audio is provided internally or externally. This provision can take various forms, such as sharing information with company members or publishing on a website. Specifically, for company members, it is provided via email or the company network, and for publishing on a website, it is provided as a streaming service or download.

[1430] Example 1

[1431] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1432] In conventional systems, data collection and preprocessing to learn a specific person's words and actions, guidelines for action, and life philosophy were often done manually, which was problematic in terms of efficiency. In addition, the performance evaluation of the trained model and the generation of responses to user questions were not sufficiently automated, resulting in a lack of accuracy and reliability. Furthermore, there were limited means to effectively provide the generated video and audio, which made it difficult to improve the user experience.

[1433] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1434] In this invention, the server includes a means for automatically collecting public information related to a specific person from the Internet, a means for preprocessing the collected data, a means for training a deep learning model using the preprocessed data, a means for evaluating the performance of the trained model, and a means for generating responses to questions from users. This makes it possible to automate a series of processes from data collection to model training, performance evaluation, and response generation, and provide an efficient and highly accurate system.

[1435] A "specific person" refers to a specific individual, and refers to information regarding their words and actions, code of conduct, life philosophy, past works, etc.

[1436] "Behavior" refers to words and actions made by a particular person, and reflects that person's characteristics and personality.

[1437] A "code of conduct" refers to the basic policies or principles by which a particular person acts, and indicates that person's values ​​and beliefs.

[1438] A "philosophy of life" refers to a person's thoughts and beliefs about life, and reflects that person's way of life and values.

[1439] "Past books" refer to books written by a particular person or about that person, which record that person's thoughts and activities.

[1440] "Public information" refers to information available on the Internet or from other public sources, including data relating to a particular person.

[1441] "Automated collection means" refers to the use of programs or algorithms to collect publicly available information relating to a particular person without manual intervention.

[1442] "Preprocessing means" refers to processes for converting collected data into a format that is easier to analyze, including data cleaning and normalization.

[1443] A "deep learning model" refers to a machine learning model that uses neural networks and has the ability to learn complex patterns and features.

[1444] "Means of evaluating performance" refers to methods for measuring the accuracy and reliability of a trained model, using evaluation metrics and test data.

[1445] The "means for generating a response" refers to a method for generating an appropriate answer to a question from a user, and creates a response using a learned model.

[1446] "Means for generating video / audio" refers to technology for creating video and audio that sounds like a specific person is speaking, using a generative AI model.

[1447] "Means of providing" refers to the method for providing the generated video and audio to the user, which is carried out through an internal system or an external platform.

[1448] "Means for publishing on a website" refers to a method for publishing the generated video and audio on a website on the Internet, providing them in a format that can be accessed by users.

[1449] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1450] This invention relates to a system that learns the words and actions, guidelines, life philosophies, past books, etc. of a specific person. This system automatically collects public information related to a specific person, preprocesses the collected data, and trains it using a deep learning model. Furthermore, the system evaluates the performance of the trained model and generates responses to questions from users.

[1451] Data collection

[1452] The server automatically collects public information related to a specific person from the Internet. Specifically, it uses Python's BeautifulSoup and Scrapy to scrape web pages and extract the necessary information. For example, it collects data such as articles, books, videos, and audio related to a specific person.

[1453] Data Preprocessing

[1454] The server preprocesses the collected data: for text data, it uses NLTK or spaCy to clean, tokenize, and normalize the text, and for audio data, it uses the Google Cloud Speech-to-Text API to convert the audio data to text, which puts the data in a format that's easier to parse.

[1455] Model Training

[1456] The server trains a deep learning model using the preprocessed data. It uses TensorFlow and PyTorch to build a model that learns a specific person's words and actions, behavioral guidelines, and life philosophy. Specifically, it trains models such as LSTM and Transformer using the collected text data as input.

[1457] Model evaluation

[1458] The server evaluates the performance of the trained model using metrics such as precision, recall, and F1 score. Specifically, it evaluates the model's predictive performance using test data and adjusts the model's hyperparameters as necessary.

[1459] Generate a response to the user

[1460] The user inputs a question about a specific person into the generative AI model from the terminal. The server receives the prompt sentence from the user and generates a response using the trained model. For example, if the user inputs "Tell me about the leadership style of a specific person," the server uses the model to generate information about the person's leadership style and provides it to the user.

[1461] Examples of concrete examples and prompts

[1462] For example, if a specific person is a "famous leader," the server will collect articles, books, videos, and audio about that leader from the Internet. The collected data will be preprocessed and the deep learning model will be used to learn the leader's words, actions, and life philosophy.

[1463] Examples of prompts to be input to a generative AI model include:

[1464] "Tell me about the leadership styles of some famous leaders."

[1465] "Describe how a notable leader led themselves to success."

[1466] "Tell me more about the life philosophies of famous leaders."

[1467] In this way, a user can obtain detailed information about a particular person.

[1468] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1469] Program processing flow

[1470] Step 1: Data collection

[1471] The server automatically collects public information related to a specific person from the Internet. Specifically, it uses Python's BeautifulSoup and Scrapy to scrape web pages and extract the necessary information. As input, it receives a list of URLs related to a specific person, and as output, it generates the collected text data and metadata.

[1472] Step 2: Data Preprocessing

[1473] The server preprocesses the collected data. For text data, it uses NLTK or spaCy to clean, tokenize, and normalize the text. For audio data, it uses Google Cloud Speech-to-Text API to convert the audio data to text. As input, it takes the collected raw data and produces preprocessed, clean text data as output.

[1474] Step 3: Model training

[1475] The server trains a deep learning model using the preprocessed data. It uses TensorFlow and PyTorch to build a model that learns a specific person's words and actions, behavioral guidelines, and life philosophy. Specifically, it trains models such as LSTM and Transformer. It receives preprocessed text data as input and generates a trained model as output.

[1476] Step 4: Model evaluation

[1477] The server evaluates the performance of the trained model. For the evaluation, it uses metrics such as precision, recall, and F1 score. Specifically, it evaluates the predictive performance of the model using test data and adjusts the model's hyperparameters as necessary. It receives the trained model and test data as input and generates the evaluation results as output.

[1478] Step 5: Accepting user questions

[1479] A user inputs a question about a specific person into the generative AI model from a terminal. The model receives the user's question as input and sends the question to the server as output.

[1480] Step 6: Response Generation

[1481] The server receives a prompt from the user and generates a response using the trained model. Specifically, the server inputs the user's question into the model and generates an appropriate answer. The server receives the user's question and the trained model as input, and generates the generated response as output.

[1482] Step 7: Providing a response

[1483] The server provides the generated response sentence to the user. Specifically, the server transmits the generated response sentence to the terminal so that the user can view it. The server receives the generated response sentence as input and displays the response sentence on the user's terminal as output.

[1484] In this way, the entire process from data collection to response generation is automated, resulting in an efficient and highly accurate system.

[1485] (Application example 1)

[1486] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1487] In modern society, there is a demand to learn the words and actions, guidelines for action, and life philosophies of specific people and apply them to daily life and work. However, there is a lack of efficient means to collect this information and provide it to users in an appropriate format. Furthermore, there is no system that automatically collects public information about specific people and uses it to provide specific advice and motivation to users. As a result, users must spend a great deal of time and effort collecting and analyzing information.

[1488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1489] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question or inquiry, a means for providing the generated video / audio internally or externally, a means for automatically collecting public information about the specific person, and a means for providing the user with advice and motivation that is useful in daily life and work based on the collected information. This allows the user to efficiently learn the philosophy and guidelines of a specific person and use them in daily life and work.

[1490] A "specific person" refers to a specific individual, and refers to information about that person's words and actions, code of conduct, life philosophy, past books, etc.

[1491] "Words and actions" refers to what a particular person says or does.

[1492] A "code of conduct" refers to the basic ideas or policies that a particular person follows when acting.

[1493] "Philosophy of life" refers to a person's basic thoughts and values ​​regarding life.

[1494] "Past books" refers to books written by a particular person or books about that person.

[1495] "Means of learning" refers to a method of learning using machine learning technology based on information such as a specific person's words and actions, code of conduct, philosophy of life, and past books.

[1496] "Question / Inquiry" refers to a question or inquiry entered by a user for a specific person.

[1497] "Means for generating video / audio" refers to means for generating video and audio that sounds like a specific trained person is speaking.

[1498] The "means for providing" refers to the means for providing the generated video and audio internally or externally.

[1499] "Means of automatically collecting public information" refers to means of automatically collecting public information such as articles, books, videos, and audio on the Internet.

[1500] "Means of providing advice and motivation" refers to means of providing users with advice and motivation that is useful in their daily lives and work, based on the collected information.

[1501] A system for implementing this invention includes means for learning the words and actions, guidelines for action, life philosophy, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person is speaking in response to an input question or inquiry, means for providing the generated video / audio internally or externally, means for automatically collecting public information about the specific person, and means for providing the user with advice and motivation that is useful in their daily life and work based on the collected information.

[1502] System Configuration

[1503] Hardware

[1504] Server: A central processing unit for collecting, learning, generating, and providing data.

[1505] Smartphone: A device for users to input questions and inquiries and receive the generated video / audio

[1506] software

[1507] Python: A programming language for data collection, machine learning, and generative processing

[1508] BeautifulSoup: A library for scraping articles and information from the internet

[1509] TensorFlow: A library for building and training deep learning models

[1510] Data processing and calculation

[1511] Data collection

[1512] The server automatically collects public information about a particular person by using BeautifulSoup to scrape information such as articles, books, videos, and audio from the Internet.

[1513] Data Preparation

[1514] Based on the collected information, we create a dataset to learn the words and actions, behavioral guidelines, and life philosophies of a particular person, which includes preprocessing and labeling the text data.

[1515] Model Building and Training

[1516] Using TensorFlow, a Long Short-Term Memory (LSTM) model is built and trained on the collected data, creating a generative AI model that mimics the behavior and course of action of a specific person.

[1517] Video / audio generation

[1518] Using the learned model, video / audio that sounds like a specific person is speaking in response to a question or inquiry entered by the user is generated.

[1519] offer

[1520] The generated video and audio are sent from the server to the user's smartphone, allowing the user to efficiently learn the philosophy and code of conduct of a particular person and apply it to their daily life and work.

[1521] Examples

[1522] When a user inputs the name of a "certain celebrity" into their smartphone, the server collects publicly available information about that person and learns from it. Then, when the user inputs a question such as "What would a particular celebrity think in this situation?", the server uses the learned model to generate video and audio that sounds like the person is speaking and sends it to the smartphone.

[1523] Examples of prompt statements

[1524] Enter the name of a specific celebrity. For example, "Specific Celebrity A" or "Specific Celebrity B".

[1525] In this way, users can learn the philosophies and codes of conduct of a particular person and apply them in their daily lives and work.

[1526] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1527] Step 1:

[1528] A user inputs the name of a specific person into a smartphone terminal. The input name is sent to a server. The input data is the name of a specific person.

[1529] Step 2:

[1530] The server automatically collects public information such as articles, books, videos, and audio on the Internet based on the input name of a specific person. Specifically, it uses BeautifulSoup to perform web scraping to obtain related text data. The input data is the name of a specific person, and the output data is the collected public information.

[1531] Step 3:

[1532] The server creates a dataset based on the collected public information to learn the words and actions, behavioral guidelines, and life philosophies of a specific person. It preprocesses the text data (tokenization, normalization, labeling, etc.) and formats the learning data. The input data is the collected public information, and the output data is the preprocessed learning data.

[1533] Step 4:

[1534] The server uses TensorFlow to build a Long Short-Term Memory (LSTM) model and trains the model based on preprocessed training data. This creates a generative AI model that mimics the words, actions, and behavior of a specific person. The input data is the preprocessed training data, and the output data is the trained generative AI model.

[1535] Step 5:

[1536] A user inputs a question or inquiry into a smartphone terminal. The input question or inquiry is sent to a server. The input data is the content of the user's question or inquiry.

[1537] Step 6:

[1538] The server uses a trained generative AI model to generate video / audio that sounds like a specific person is speaking in response to a question or inquiry input by a user. The input data is the user's question or inquiry content and the trained generative AI model, and the output data is the generated video / audio.

[1539] Step 7:

[1540] The server transmits the generated video / audio to the smartphone terminal. The user can receive and view the generated video / audio on the smartphone terminal. The input data is the generated video / audio, and the output data is the video / audio displayed on the user's smartphone terminal.

[1541] In this way, users can learn the philosophies and code of conduct of a particular person and apply them to their daily lives and work.

[1542] Example 2

[1543] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1544] In conventional systems, it was difficult to generate video and audio that sounded like a specific person was actually answering a user's question or inquiry. In addition, the quality of the generated video and audio was low, making it impossible to provide a realistic experience to the user. Furthermore, there was a lack of means to learn the words, actions, and guidelines of a specific person, so the generated answers lacked reliability and consistency.

[1545] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1546] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for analyzing input text using natural language processing technology for input questions and inquiries, a means for generating a prompt sentence based on the analysis result, a means for inputting the prompt sentence into a generation AI model and generating an answer text, a means for generating audio in the voice of a specific person using speech synthesis technology from the generated answer text, a means for generating an image of a specific person speaking using video generation technology based on the generated audio, and a means for providing the generated video / audio internally or externally. This makes it possible to generate high-quality video and audio as if a specific person were actually answering questions and inquiries from users, providing a realistic experience.

[1547] A "specific person" is a specific individual designated by the user, and is a person whose words and actions, code of conduct, philosophy of life, past books, etc. are to be studied.

[1548] "Behavior" refers to the words and actions taken by a particular person and reflects that person's characteristics and personality.

[1549] A "code of conduct" is a standard or policy of behavior that a particular person should follow, and indicates that person's values ​​and beliefs.

[1550] A "philosophy of life" is a person's thoughts and beliefs about life, which reflect that person's way of life and values.

[1551] "Past books" refer to books written by a particular person or books related to that person, and are materials for learning about that person's ideas and knowledge.

[1552] "Natural language processing technology" refers to technology that enables computers to understand and analyze human language, analyzing text and extracting meaning.

[1553] A "prompt" is a document to be input into a generative AI model, and is an instruction text for generating answers to users' questions or inquiries.

[1554] A "generative AI model" is a model that uses artificial intelligence to generate text or speech, and generates answers based on prompts.

[1555] "Speech synthesis technology" refers to technology that converts text into speech and is used to generate speech in the voice of a specific person.

[1556] "Image generation technology" refers to technology that generates images based on audio or text, and is used to create images of a specific person speaking.

[1557] "Means for providing internally or externally" refers to means for storing the generated video and audio within the system, or for providing them to an external user or system.

[1558] The present invention is a system for generating video and audio that sounds as if a specific person is actually answering a question or inquiry from a user. This system is implemented using the following hardware and software.

[1559] Hardware and Software Used

[1560] Hardware: Server equipped with a high-performance GPU

[1561] Software: Natural Language Processing (NLP) libraries (e.g. spaCy, Transformers), voice synthesis software (e.g. Tacotron 2), video generation software (e.g. DeepFake technology)

[1562] Specific operation of the system

[1563] When the server receives a question or request for advice from a user, it first analyzes the input text using natural language processing technology. The analysis includes grammatical analysis and keyword extraction. It then generates a prompt sentence based on the analysis results. For example, if a user enters, "I haven't been able to sleep at night lately. What should I do?", the following prompt sentence will be generated:

[1564] Example prompt:

[1565] "Health Advice: I haven't been able to sleep at night lately, what should I do? Let your doctor answer this question."

[1566] The server inputs this prompt sentence into a generative AI model (e.g., GPT-3) to generate an answer text. The generated answer text might look something like this:

[1567] Example of a generated answer:

[1568] "There are several possible reasons why you can't sleep at night. First, it's important to make time to relax before going to bed. For example, taking a warm bath or doing some light stretching is effective. It's also important to limit your caffeine intake."

[1569] Next, the server inputs the generated response text into speech synthesis software to generate audio in a specific person's voice. For example, audio is generated in the voice of a doctor. Then, based on the generated audio, video generation software is used to generate video of the specific person speaking. Finally, the generated audio and video are integrated into a single video file.

[1570] Examples

[1571] For example, if a user enters "I haven't been able to sleep at night lately. What should I do?", the process will proceed as follows:

[1572] 1. User input: The user uses a terminal to enter a question.

[1573] 2. Text analysis: The server uses an NLP library to extract keywords such as "I can't sleep at night" and "What should I do?"

[1574] 3. Prompt generation: The server generates a prompt such as "Health consultation: I haven't been able to sleep at night lately. What should I do? Please have a doctor answer this question."

[1575] 4. Input to the generative AI model: The server inputs the prompt sentence into GPT-3.

[1576] 5. Answer generation: GPT-3 generates the answer, "There are several possible reasons why you can't sleep at night. First, it is important to take time to relax before going to bed. For example, taking a warm bath or doing some light stretching can be effective. It is also important to limit your caffeine intake."

[1577] 6. Speech synthesis: The server uses Tacotron 2 to generate speech in the doctor’s voice.

[1578] 7. Video Generation: The server uses DeepFake technology to generate a video of the doctor speaking.

[1579] 8. Audio and video integration: The server integrates the generated audio and video.

[1580] 9. Result delivery: The server sends the consolidated video file to the user's device, where the user watches the video.

[1581] In this way, the user can experience high quality video and audio as if a specific person were actually answering the question.

[1582] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1583] Step 1:

[1584] Users access the system using a terminal and input their questions or inquiries. The input text is then sent to the server.

[1585] Input: User's question or concern (e.g. "I haven't been able to sleep at night lately. What should I do?")

[1586] Output: The input text sent to the server.

[1587] Step 2:

[1588] The server analyzes the received input text using a natural language processing (NLP) library (e.g., spaCy), which includes grammar analysis and keyword extraction.

[1589] Input: The input text sent by the user

[1590] Data processing: grammatical analysis, keyword extraction

[1591] Output: Analysis results (e.g. keywords such as "I can't sleep at night" and "What should I do")

[1592] Step 3:

[1593] The server generates a prompt sentence based on the analysis results. The prompt sentence is an instruction sentence to be input to the generative AI model.

[1594] Input: Analysis results

[1595] Data processing: Prompt generation

[1596] Output: Prompt text (e.g. "Health question: I've been having trouble sleeping at night lately. What should I do? Please have your doctor answer this question.")

[1597] Step 4:

[1598] The server inputs the generated prompt sentence into a generative AI model (e.g., GPT-3) to generate an answer text.

[1599] Input: prompt statement

[1600] Data computation: Generative AI model generates answers

[1601] Output: Answer text (Example: "There are several possible reasons why you can't sleep at night. First, it's important to take time to relax before going to bed. For example, taking a warm bath or doing some light stretching can be effective. It's also important to limit your caffeine intake.")

[1602] Step 5:

[1603] The server inputs the generated answer text into speech synthesis software (e.g., Tacotron 2) to generate a voice in the voice of a specific person.

[1604] Input: Answer text

[1605] Data Calculation: Voice Synthesis

[1606] Output: Audio data (e.g. generated audio of a doctor's voice)

[1607] Step 6:

[1608] The server uses video generation software (e.g., DeepFake technology) based on the generated audio to generate a video of a specific person speaking.

[1609] Input: Audio data

[1610] Data Calculation: Image Generation

[1611] Output: Video data (e.g. video of a doctor speaking)

[1612] Step 7:

[1613] The server combines the generated audio and video into a single video file.

[1614] Input: Audio data, video data

[1615] Data processing: Audio and video integration

[1616] Output: A merged video file

[1617] Step 8:

[1618] The server transmits the combined video file to the user's terminal, and the user watches the video.

[1619] Input: merged video file

[1620] Output: Video file sent to user's device

[1621] The above is the specific processing flow of the program of this system.

[1622] (Application example 2)

[1623] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1624] In conventional customer service in brick-and-mortar stores, when a customer has a question or inquiry, a sales clerk is required to respond, and because this depends on the number and skills of the sales clerks, the response may be delayed or an insufficient answer may be provided. Furthermore, if a specific sales clerk is not available or there are only a limited number of sales clerks with specialized knowledge, customer satisfaction may decrease. To solve these problems, there is a demand for a system that can generate video and audio that sounds like a specific person is speaking, and have a virtual sales clerk serve customers in a brick-and-mortar store.

[1625] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for learning the words and actions, guidelines for action, philosophy of life, past books, etc. of a specific person, a means for generating video / audio that sounds like the learned specific person is speaking in response to an input question / inquiry, etc., a means for providing the generated video / audio internally or externally, a means for generating video / audio of a virtual store clerk answering a customer's question in a real store, and a means for providing the generated video / audio to a smartphone. This makes it possible for a customer in a real store to get a quick and accurate answer from a virtual store clerk using a smartphone.

[1626] A "specific person" refers to an individual whose words and actions, code of conduct, life philosophy, past books, etc. are the subject of study.

[1627] "Speech and behavior" refers to the words and actions of a particular person.

[1628] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[1629] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[1630] "Past books" refers to books and publications written by a specific person.

[1631] "Means of learning" refers to methods and techniques for collecting and analyzing a specific person's words and actions, code of conduct, philosophy of life, past books, etc. as data.

[1632] "Question / Inquiry" refers to any inquiry or request for advice that a User enters into the System.

[1633] "Means for generating video / audio" refers to technology or methods for producing video and audio that sounds as if a specific trained person is actually speaking.

[1634] "Means of providing" refers to the methods and technologies for delivering the generated video and audio to users internally or externally.

[1635] "Brick and mortar store" refers to a commercial establishment that exists in a physical location.

[1636] A "virtual store clerk" refers to a virtual store clerk created to serve customers in a physical store.

[1637] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and run applications.

[1638] A system for implementing this invention includes means for learning the words and actions, code of conduct, philosophy of life, past books, etc. of a specific person, means for generating video / audio that sounds like the specific person learned is speaking in response to input questions, inquiries, etc., means for providing the generated video / audio internally or externally, means for generating video / audio of a virtual store clerk answering customer questions in a physical store, and means for providing the generated video / audio to a smartphone.

[1639] Hardware and Software Used

[1640] Hardware: Smartphones, servers

[1641] Software: Python, OpenAI API, OpenCV, pyttsx3

[1642] Data processing and calculation

[1643] 1. The server collects data such as a specific person's words and actions, guidelines, life philosophy, past books, etc., and trains the system using natural language processing technology.

[1644] 2. The user uses their smartphone to enter their question or inquiry.

[1645] 3. The server uses the OpenAI API to generate answers to the questions entered, as if the specific person were actually answering them.

[1646] 4. The server converts the generated answer into an audio file using pyttsx3.

[1647] 5. The server uses OpenCV to combine the audio file with the pre-recorded video of the store clerk to generate the final video.

[1648] 6. The server provides the generated video / audio to the smartphone.

[1649] Examples

[1650] For example, when a user enters a question such as "What are the features of this product?" at a physical store, the server performs the following processing.

[1651] 1. The server uses the OpenAI API to generate an answer to the question, "What are the features of this product?"

[1652] 2. The generated answer will be, "This product uses the latest technology and is very high performance."

[1653] 3. The server converts this response into an audio file using pyttsx3.

[1654] 4. The server uses OpenCV to combine this audio file with the pre-recorded video of the store clerk to generate the final video.

[1655] 5. Users can view and hear the virtual store clerk on their smartphones.

[1656] Examples of prompt statements

[1657] Q: What are the features of this product?

[1658] A: This product uses the latest technology and is very powerful.

[1659] In this way, customers can use their smartphones to get quick and accurate answers from virtual store clerks in physical stores.

[1660] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1661] Step 1:

[1662] The user inputs a question or inquiry using a smartphone. The input question is sent to the server as text data.

[1663] Step 2:

[1664] The server sends the received question to the OpenAI API and uses a generative AI model to generate an answer to the question. The input is the user's question text and the output is the generated answer text.

[1665] Step 3:

[1666] The server converts the generated response text into an audio file using pyttsx3. The input is the response text, and the output is the audio file (e.g., response.mp3).

[1667] Step 4:

[1668] The server uses OpenCV to combine the pre-recorded video of the store attendant with the audio file to generate the final video. The input is the pre-recorded video file and the generated audio file, and the output is the final video file (e.g., response_video.mp4).

[1669] Step 5:

[1670] The server provides the generated video / audio to a smartphone. The input is the final video file, and the output is the video / audio played on the smartphone.

[1671] Step 6:

[1672] The user sees the video and audio of the virtual store assistant on his / her smartphone. The input is the video / audio file provided by the server, and the output is the user's visual and auditory reception of the information.

[1673] Example 3

[1674] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1675] In conventional systems, the means to generate the knowledge and experience of a specific person as video and audio and share it both inside and outside the company were limited. It was also difficult to efficiently store the generated video and audio and provide it at the appropriate time. This made it impossible to make the most of the knowledge and experience of a specific person.

[1676] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1677] In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the specific person who has learned is speaking in response to an input question / consultation, etc., a means for inputting a prompt sentence into the generative AI model, a means for the generative AI model to generate video / audio, a means for saving the generated video / audio, a means for providing the generated video / audio internally or externally, a means for providing the generated video / audio to company members, and a means for publishing the generated video / audio on a website. This makes it possible to efficiently generate the knowledge and experience of a specific person as video and audio and share it both inside and outside the company.

[1678] A "specific person" refers to an individual with specific knowledge or experience.

[1679] "Speech and behavior" refers to the words and actions of a particular person.

[1680] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[1681] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[1682] "Past books" refers to books and documents written by a specific person.

[1683] "Means of learning" refers to a method for incorporating a particular person's words and actions, guidelines for behavior, life philosophy, past books, etc. into the system.

[1684] "Questions and inquiries" refers to inquiries or requests for advice that users input into the system.

[1685] "Means for generating video / audio" refers to a method for creating video and audio that sounds like a specific person is speaking based on an input question or inquiry.

[1686] A "generative AI model" refers to a model that uses artificial intelligence to generate images and sounds.

[1687] A "prompt sentence" refers to an input sentence that gives instructions to a generative AI model.

[1688] "Means of storage" refers to the method of storing the generated video and audio as data.

[1689] "Means of providing" refers to the method of distributing the generated video and audio internally or externally.

[1690] "Means of providing to internal members" refers to the method of sharing the generated video and audio with internal stakeholders.

[1691] "Means of publishing on a website" refers to a method of widely publishing the generated video and audio on the Internet.

[1692] This invention relates to a system that generates a specific person's knowledge and experience as video and audio and shares them both inside and outside the company. This system includes a means for learning the specific person's words and actions, guidelines for action, life philosophy, past books, etc., a means for generating video / audio that sounds like the specific person is speaking in response to an input question or consultation, a means for inputting a prompt sentence into a generative AI model, a means for the generative AI model to generate video / audio, a means for saving the generated video / audio, a means for providing the generated video / audio internally or externally, a means for providing the generated video / audio to company members, and a means for publishing the generated video / audio on a website.

[1693] The user inputs a prompt for the generative AI model using a dedicated application or web interface. For example, the user might input, "Please create a promotional video for our new product." The device then sends the input prompt to the server using the HTTPS protocol.

[1694] The server receives the prompt text sent from the terminal and stores it in a temporary database. The server then inputs the stored prompt text into the generative AI model and starts the generation process. Examples of generative AI models include OpenAI's GPT-4 and DALL-E. The generative AI model generates video and audio based on the prompt text. This generation process can take anywhere from a few seconds to a few minutes.

[1695] The generated video and audio are stored in the server's storage. For example, cloud storage such as Amazon S3 or Google Cloud Storage is used as the storage location. The server uploads the saved video and audio to an internal shared folder to provide it to company members, or makes it available on an internal web portal.

[1696] Furthermore, the server publishes the generated video and audio on a website, using a website management system (e.g., WordPress or Joomla) to publish the content.

[1697] As a concrete example, consider the case where a user inputs a prompt saying, "I want you to generate an interview video about a specific project." When this prompt is sent to the server, the server inputs this prompt into the generative AI model. The generative AI model generates an interview video about the project and returns it to the server. The server uploads the generated video to a shared folder within the company so that members within the company can access it. The server can also publish the video on an external website to share it widely.

[1698] Examples of prompts include:

[1699] "I want you to generate interview footage for a specific project."

[1700] "I want you to create a video to introduce our new product."

[1701] "I want you to generate a commentary audio about a specific technology."

[1702] In this way, the user can utilize the generative AI model to generate video and audio according to specific needs and share them inside and outside the company. The flow of the specification process in the third embodiment will be described with reference to FIG.

[1703] Step 1:

[1704] The user inputs a prompt for the generative AI model using a dedicated application or web interface. For example, the user might input, "Please create a promotional video for our new product." The input prompt is temporarily stored in the device's memory.

[1705] Step 2:

[1706] The terminal sends the input prompt text to the server using the HTTPS protocol. The prompt text is encrypted before being sent. The input is the prompt text, and the output is a notification of completion of transmission to the server.

[1707] Step 3:

[1708] The server receives the prompt text sent from the terminal and stores it in a temporary database. The input is the prompt text, and the output is a notification that the text has been saved to the database.

[1709] Step 4:

[1710] The server inputs the saved prompt sentence into the generative AI model and starts the generation process. The input is the prompt sentence, and the output is a notification to the generative AI model that the input has been completed.

[1711] Step 5:

[1712] A generative AI model generates video and audio based on a prompt. This generation process can take anywhere from a few seconds to a few minutes. The input is the prompt, and the output is the generated video and audio.

[1713] Step 6:

[1714] The server receives the video and audio returned from the generative AI model and stores them in the server's storage. The storage location can be cloud storage such as Amazon S3 or Google Cloud Storage. The input is the generated video and audio, and the output is a notification that the storage has been completed.

[1715] Step 7:

[1716] The server uploads the stored video and audio to a shared folder within the company to provide them to company members, or publishes them on a web portal for internal use only. The input is the stored video and audio, and the output is a notification to company members that the video and audio have been provided.

[1717] Step 8:

[1718] The server publishes the generated video and audio on a website. At this time, the content is published using a website management system (e.g., WordPress or Joomla). The input is the saved video and audio, and the output is a notification on the website that publication has been completed.

[1719] (Application example 3)

[1720] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1721] In modern society, there is a demand to widely share the knowledge and experience of specific experts and experienced people. However, there is a lack of means to efficiently collect this knowledge and experience, generate it as video or audio, and provide it to in-house members and general users. This makes it difficult to effectively communicate specialized knowledge and experience.

[1722] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a means for learning the words and actions, guidelines, life philosophy, past books, etc. of a specific person, a means for generating video / audio that sounds like the learned specific person is speaking in response to an input question / consultation, etc., a means for providing the generated video / audio internally or externally, a means for collecting knowledge and experience of specific experts and experienced people in text format, a means for creating a prompt sentence to be input to the generation AI model based on the collected text data, a means for inputting the prompt sentence to the generation AI model and generating video and audio, and a means for publishing the generated video and audio to in-house members or on a website. This makes it possible to efficiently collect the knowledge and experience of specific experts and experienced people, generate it as video and audio, and share it widely.

[1723] A "specific person" refers to a specific expert or person with experience whose knowledge or experience is important.

[1724] "Speech and behavior" refers to the words and actions of a particular person.

[1725] A "code of conduct" refers to a standard or policy of behavior that a particular person should follow.

[1726] "Philosophy of life" refers to a particular person's way of thinking and values ​​regarding life.

[1727] "Past books" refers to books and documents written in the past by a particular person.

[1728] "Means of learning" refers to methods or techniques for making the system learn a particular person's words and actions, code of conduct, philosophy of life, past books, etc.

[1729] "Input questions and inquiries" refers to questions and inquiries that u...

Claims

1. A means to study the words and actions of a particular person, their code of conduct, their life philosophy, and past books, A means for generating video and audio that sounds like a specific person who has been trained is speaking in response to an input question or inquiry; A means for providing the generated video and audio internally or externally; A means for preprocessing the learned data; A means for training a deep learning model using the preprocessed data; A means for evaluating the performance of the trained model; and means for generating responses to questions or inquiries from a user; Including, The means for generating a response is a system that generates a response using a generative AI model based on a prompt sentence that combines the user's emotional state and the thoughts of the specific person.

2. The system of claim 1 , wherein the means for training includes means for automatically collecting public information of a particular person.

3. The system of claim 1 , wherein the means for providing includes means for publishing the generated video and audio on a website.

Images