System

Abstract

To provide a system.SOLUTION: The system includes: learning means for learning a speech pattern of a user by using a generation AI model; generation means for generating a clone of the user on the basis of the learned speech pattern; means for updating the speech pattern through a chat between the generated clone and the user; means for inputting a specific prompt sentence into the generation AI model by using an emotion engine for recognizing the emotion of the user and the generation AI model, thereby generating a response of the clone according to the emotion; and means for using the generated response of the clone for a conversation with a third person.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] There is no system that allows a user to create a clone with the user's own speech patterns, have the clone have the same conversation as the user, and further have a conversation with a third party on the user's behalf. [Means for solving the problem]

[0005] The system of the present disclosure includes a learning means that uses a generative AI model to learn a user's speech patterns, a generation means that generates a clone of the user based on the learned speech patterns, a means that updates the speech patterns through a chat between the generated clone and the user, an emotion engine that recognizes the user's emotions, a generation means that uses a generative AI model to generate a response of the clone that corresponds to the emotion by inputting a specific prompt sentence into the generative AI model, and a means that uses the generated response of the clone in a conversation with a third party. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One embodiment of the present invention is a system in which a user makes statements through a messenger app. This system learns the user's speech patterns and generates a clone of the user based on the learning results. The generated clone updates its speech patterns through chats with the user. Specifically, if the user says "Good morning," the clone also learns to say "Good morning."

[0029] "Example 2"

[0030] Furthermore, the clone can have a conversation similar to that of the user. For example, if the user asks, "What's the weather like today?", the clone will learn to respond, "It's sunny."

[0031] "Example 3"

[0032] The clone can also converse with third parties on behalf of the user. Specifically, the user can leave conversations with people the clone does not like to talk to. Because the clone has the same speech patterns as the user, third parties may mistake the clone for the user.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user makes a comment through the messenger app.

[0036] Step 2: The system learns the user's speech patterns.

[0037] Step 3: Generate a clone of the user based on the learning results.

[0038] Step 4: The generated clone updates its speech patterns through chat with the user. Specifically, if the user says "Good morning," the clone also learns to say "Good morning."

[0039] "Example 2"

[0040] Step 1: The clone has a similar conversation to the user.

[0041] Step 2: Specifically, when a user asks, "What's the weather like today?" the clone learns to answer, "It's sunny."

[0042] "Example 3"

[0043] Step 1: The clone talks to a third party on behalf of the user.

[0044] Step 2: Specifically, the user can leave conversations with people they don't like to the clone.

[0045] Step 3: Because the clone has similar speech patterns to the user, a third party may mistakenly believe that the clone is the user.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] Conventional messenger applications lacked the ability to learn a user's speech patterns and generate a clone of the user based on them. This created a need for a system that could automatically respond when the user was absent. It was also difficult to update the user's speech patterns in real time and achieve more natural conversations.

[0049] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for receiving user utterances, means for saving the received utterances in a database, means for analyzing utterance patterns based on the saved utterance data, means for generating a clone of the user based on the analysis results, and means for updating the utterance patterns of the clone through chat between the generated clone and the user. This makes it possible to learn the user's utterance patterns, generate a clone, and update the utterance patterns in real time. The server may include learning means for learning the user's utterance patterns using a generative AI model.

[0050] "User" means an individual or entity that uses the System to make statements through a messenger application.

[0051] A "statement" is a communication such as a text message or a voice message that a user sends through a messenger application.

[0052] "Means for receiving" refers to the function or process by which a user's comments are sent to a server and the server receives the comments.

[0053] A "database" is an information system that stores user utterance data and manages it so that it can be accessed and analyzed later.

[0054] "Means for saving" refers to the functions and processes for recording and storing received user comments in a database.

[0055] A "statement pattern" is a collection of data obtained by analyzing the characteristics, tendencies, frequency, context, etc. of a user's statements.

[0056] The "analysis means" refers to a function or process for extracting and analyzing a user's speech patterns based on the saved speech data.

[0057] A "clone" is a virtual agent that is generated based on the user's speech patterns and can make speeches and respond in the same way as the user.

[0058] "Means for generating" refers to the functions and processes for creating a clone of a user based on the analysis results.

[0059] "Chat" refers to the interaction or communication between a user and a clone via a messenger application.

[0060] The "means for updating" refers to a function or process by which the clone learns new speech patterns through chat with the user and improves the clone's speech patterns in real time.

[0061] The present invention is a system in which a user makes statements through a messenger application. The system learns the user's statement patterns and generates a clone of the user based on the learning results. The generated clone updates its statement patterns through chat with the user. The server may include a learning means that uses a generative AI model to learn the user's statement patterns. The server may include a generation means that generates a clone of the user based on the learned statement patterns. The server may include a means for updating the statement patterns through chat between the generated clone and the user.

[0062] Hardware and software used

[0063] Hardware

[0064] Server: A server equipped with a high-performance CPU and GPU

[0065] Device: The smartphone or PC used by the user

[0066] software

[0067] Generative AI models: for example, GPT-4 (registered trademark) from OpenAI (registered trademark)

[0068] Messenger applications: for example, Slack or MICROSOFT® TEAMS®

[0069] Data processing and calculation

[0070] server

[0071] The server receives the user's utterances and stores them in a database. Based on the stored utterance data, the generative AI model is used to analyze the user's utterance patterns. Based on the analysis results, a clone of the user is generated. The generated clone updates its utterance patterns through chat with the user. The server may include a generation means for generating a clone of the user based on the learned utterance patterns. The server may include a means for updating the utterance patterns through chat between the generated clone and the user.

[0072] Terminal

[0073] The device transmits the user's comments to the server in real time. When a user makes a comment through a messenger application, a data packet containing the comment and the user ID is generated and sent to the server.

[0074] User

[0075] A user uses a messenger application to type and send a text message, for example, the user says "Good morning."

[0076] Specific examples

[0077] 1. A user says "Good morning" in a messenger application.

[0078] 2. The device sends this message to the server.

[0079] 3. The server receives the message and stores the message "Good morning" in the database.

[0080] 4. The server uses the generative AI model to analyze the utterance pattern of "Good morning."

[0081] 5. The server generates a clone based on the analysis results.

[0082] 6. The clone replies to the user with "Good morning" and updates its speech patterns through chat with the user.

[0083] Prompt Sentence Examples

[0084] "Based on what the user says in the messenger application, learn the user's speech patterns and generate a clone. For example, if the user says 'Good morning,' the clone should also say 'Good morning.'"

[0085] In this way, a system is constructed that learns a user's speech patterns, generates clones, and updates the speech patterns in real time. The server may include a learning means that uses a generative AI model to learn the user's speech patterns. The server may include a generation means that generates a clone of the user based on the learned speech patterns. The server may include a means for updating the speech patterns through a chat between the generated clone and the user.

[0086] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0087] Step 1:

[0088] A user speaks in a messenger application. The user uses the messenger application to type and send a text message. For example, the user speaks "Good morning." This utterance becomes input data.

[0089] Step 2:

[0090] The device sends the user's comments to the server. The device sends the user's comments to the server in real time. At this time, a data packet containing the comment content and the user ID is generated. The input data is the user's comment and user ID, and the output data is the data packet sent to the server.

[0091] Step 3:

[0092] The server receives the message and stores it in the database. The server receives the data packet sent from the terminal. The received message content and user ID are stored in the database. For example, the following record is added to the database:

[0093] | User ID | Post content | Timestamp |

[0094] |--------|-----------|--------------------------|

[0095] | 12345 | Good Morning | 2023-10-01 08:00:00 |

[0096] The input data are the received data packets and the output data are the records stored in the database.

[0097] Step 4:

[0098] The server analyzes speech patterns using a generative AI model. Based on the saved speech data, the server analyzes the user's speech patterns using a generative AI model (e.g., OpenAI GPT-4). The analysis results include extracting the characteristics, frequency, and context of the user's speech. The input data is the speech data saved in the database, and the output data is the analysis results.

[0099] Step 5:

[0100] The server generates a clone based on the analysis results. The server generates a clone of the user based on the analysis results. This clone is designed to mimic the user's speech patterns. For example, if the user frequently says "Good morning," the clone is set to similarly say "Good morning." The input data is the analysis results, and the output data is the generated clone.

[0101] Step 6:

[0102] The clone updates its speech patterns through chat with the user. The clone updates its speech patterns in real time through chat with the user. For example, if the user says "Good morning," the clone will also reply "Good morning." Through this interaction, the clone learns the user's new speech patterns and begins to generate more accurate responses. The input data is the content of the chat with the user, and the output data is the clone's updated speech patterns.

[0103] (Application example 1)

[0104] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] While existing messenger applications can learn a user's speech patterns and generate user clones, they lack the functionality to recommend content based on the user's preferences, making it difficult to efficiently provide the information and entertainment that users desire.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0107] In this invention, the server includes means for learning a user's utterance patterns, means for generating a clone of the user based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, and means for recommending appropriate content based on the user's utterances, thereby enabling content to be recommended based on the user's preferences.

[0108] "User" means an individual or organization that uses the system.

[0109] "Speech patterns" refer to the characteristics and tendencies of statements made by users through messenger applications.

[0110] "Means of learning" refers to algorithms and programs that analyze users' speech patterns and extract their characteristics.

[0111] A "clone" is a virtual alter ego of a user that is generated based on the user's speech patterns.

[0112] "Means of generation" refers to algorithms or programs for creating clones based on learned speech patterns.

[0113] "Means for updating" refers to algorithms or programs for continuously improving the speech patterns of the generated clones through chats with users.

[0114] "Content" refers to information and entertainment such as movies, music, and news.

[0115] "Recommendation methods" refer to algorithms or programs that select and present appropriate content based on user comments.

[0116] A system for implementing this invention learns a user's speech patterns, generates a clone of the user based on the learned speech patterns, updates the clone's speech patterns through chat between the generated clone and the user, and further recommends appropriate content based on the user's speech.

[0117] System configuration

[0118] The system consists of the following main components:

[0119] 1. User device: The user uses the messenger application on a device such as a smartphone or tablet.

[0120] 2. Server: Learns the user's speech patterns, generates clones, updates the clones' speech patterns, and performs processing to recommend content.

[0121] 3. Generative AI model: Used to learn user speech patterns, generate clones, and recommend content.

[0122] Program processing

[0123] The server receives utterance data sent from the user's device and learns the utterance patterns using a generative AI model. Based on the learned utterance patterns, it generates a clone of the user. The generated clone continues to update its utterance patterns through chat with the user. Furthermore, it recommends appropriate content based on the user's utterances.

[0124] Hardware and software used

[0125] Hardware: smartphones, tablets, servers

[0126] Software: Python, OpenAI API

[0127] Data processing and calculation

[0128] Data processing: Analyze user speech data and extract speech patterns.

[0129] Data Computing: Generative AI models are used to generate clones and recommend content based on speech patterns.

[0130] Specific examples

[0131] For example, if a user says, "What movies do you recommend these days?", the server receives this utterance and uses a generative AI model to learn the user's utterance patterns. Next, it generates a clone based on the learned utterance patterns and updates the clone's utterance patterns through chat with the user. Finally, it recommends appropriate movies based on the user's utterances.

[0132] Prompt Sentence Examples

[0133] This is an assistant that learns the user's speech patterns and recommends appropriate content. Please recommend appropriate content based on the user's speech below.

[0134] User: What recent movies do you recommend?

[0135] assistant:

[0136] In this way, content can be recommended based on the user's preferences.

[0137] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0138] Step 1:

[0139] The user's terminal inputs the user's message through a messenger application.

[0140] Input: User utterance (e.g., "What are some recent movie recommendations?")

[0141] Output: Speech data

[0142] Specific operation: A user inputs a message into a messenger application, and the message data is sent to a server.

[0143] Step 2:

[0144] The server analyzes the received speech data and learns the user's speech patterns.

[0145] Input: Speech data

[0146] Output: Learned speech patterns

[0147] Specific operation: The server analyzes the speech data and extracts speech patterns using a generative AI model.

[0148] Step 3:

[0149] The server generates a clone of the user based on the learned speech patterns.

[0150] Input: Learned speech patterns

[0151] Output: Clone the user

[0152] How it works: The server uses the generative AI model to generate clones based on learned speech patterns.

[0153] Step 4:

[0154] The server updates the clone's speech patterns through chat between the clone and the user.

[0155] Input: User's new utterance, clone's utterance pattern

[0156] Output: Updated clone's utterance pattern

[0157] Specific operation: Every time the user makes a new statement, the server analyzes the statement and updates the clone's statement pattern.

[0158] Step 5:

[0159] The server recommends appropriate content based on the user's comments.

[0160] Input: User utterances, updated clone utterance patterns

[0161] Output: Recommended content

[0162] Specific operation: The server analyzes the user's comments, uses a generative AI model to select appropriate content, and recommends it to the user.

[0163] Step 6:

[0164] The user terminal displays the recommended content.

[0165] Input: Recommended content

[0166] Output: The displayed content

[0167] Specific operation: The user terminal displays the recommended content received from the server on the messenger application.

[0168] Example 2

[0169] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0170] Conventional systems have had difficulty accurately learning a user's speech patterns and generating a clone that can converse in the same way as the user. Furthermore, when the generated clone converses with a third party on behalf of the user, it sometimes fails to make appropriate remarks. This limits the usefulness of the clone as a substitute for the user.

[0171] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user utterance data, means for preprocessing the collected utterance data, means for training a generative AI model using the preprocessed data, means for generating a user clone using the trained generative AI model, and means for updating the clone's utterance pattern through chat between the generated clone and the user. This makes it possible to accurately learn the user's utterance pattern and generate a clone that can converse in a similar manner to the user. Furthermore, the generated clone can make appropriate utterances even when conversing with a third party on behalf of the user.

[0172] "User utterance data" is text data of conversations and utterances made by the user in the past.

[0173] The "collection means" refers to a method or device for acquiring and storing user utterance data.

[0174] A "preprocessing means" is a method or device for removing unnecessary information from collected speech data and arranging the text into a consistent format.

[0175] A "generative AI model" is an artificial intelligence model that learns a user's speech patterns and generates a clone that can converse in a similar way to the user.

[0176] "Means for training" refers to a method or device for training a generative AI model to learn user speech patterns using preprocessed data.

[0177] A "clone" is a virtual entity that uses a generative AI model to mimic a user's speech patterns and can have a conversation similar to that of the user.

[0178] "Means for generating" refers to a method or device for creating a clone of a user using a trained generative AI model.

[0179] The "updating means" refers to a method or device for continuously improving the speech patterns of the clone through chats between the generated clone and the user.

[0180] The present invention relates to a system that learns a user's speech patterns and generates a clone that can converse in a similar manner to the user. A specific embodiment of this system will be described below.

[0181] The server uses the API of the chat application used by the user to collect user utterance data. For example, it acquires a log of a user asking, "What's the weather like today?" This utterance data is saved in text format.

[0182] The server then preprocesses the collected speech data by removing unnecessary information (such as advertisements or system messages) and normalizing the text (such as converting uppercase to lowercase and removing special characters). This preprocessing ensures that the data is in a consistent format.

[0183] The preprocessed data is input into a generative AI model (e.g., OpenAI's GPT-3 (registered trademark)). The server uses this data to train the generative AI model. During this process, the model learns the user's speech patterns and acquires the ability to generate appropriate answers to the user's questions.

[0184] After the learning is complete, the server uses the trained generative AI model to generate a clone of the user. This clone can then have a conversation similar to the user. For example, if the user asks, "What's the weather like today?", the clone will respond, "It's sunny."

[0185] The terminal provides an interface through which the user can converse with the clone. The user types a question, and the clone responds with an appropriate answer. The interface also provides a means for the user to update the clone's speech patterns through conversation.

[0186] As a concrete example, the following prompt sentence can be input to a generative AI model:

[0187] Example prompt sentence:

[0188] User: What's the weather like today?

[0189] Crone: It's sunny.

[0190] Based on this prompt, the generative AI model generates appropriate answers to the user's questions, allowing the clone to have a conversation similar to that of the user.

[0191] This system can accurately learn a user's speech patterns and generate a clone that can converse in the same way as the user.The clone can also make appropriate remarks when conversing with a third party on behalf of the user.

[0192] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0193] Step 1: Data collection

[0194] The server collects user utterance data. Specifically, it obtains past conversation logs through the API of the chat application used by the user. These logs contain comments and questions made by the user. The input is the conversation log obtained from the chat application API, and the output is the utterance data saved in text format.

[0195] Step 2: Data Preprocessing

[0196] The server preprocesses the collected utterance data. Specifically, it performs the following processes.

[0197] Eliminate unnecessary information: Remove information that is not relevant to the conversation, such as advertisements and system messages.

[0198] Text normalization: Formatting text consistently, such as converting uppercase to lowercase and removing special characters.

[0199] The input is collected utterance data, and the output is preprocessed text data in a consistent format.

[0200] Step 3: Model training

[0201] The server inputs the preprocessed data into a generative AI model to learn the user's speech patterns. Specifically, the server supplies the preprocessed data to a generative AI model (e.g., OpenAI's GPT-3) so that the model can generate appropriate answers to the user's questions. The input is preprocessed text data, and the output is a trained generative AI model.

[0202] Step 4: Clone Generation

[0203] The server uses a trained generative AI model to generate a user clone. Specifically, the model is fed with prompts based on the user's speech patterns, and the clone generates appropriate responses. The input is the trained generative AI model and the prompts, and the output is a user clone.

[0204] Step 5: Talk to the clone

[0205] The terminal provides an interface that allows the user to converse with the clone. When the user inputs a question, the clone responds with an appropriate answer. For example, if the user inputs "What's the weather like today?", the clone responds with "It's sunny." The input is the user's question, and the output is the clone's answer.

[0206] In this way, it is possible to accurately learn a user's speech patterns and generate a clone that can converse in the same way as the user. Furthermore, the generated clone can also make appropriate statements when conversing with a third party on behalf of the user.

[0207] (Application example 2)

[0208] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0209] Conventional personal assistant systems have difficulty learning a user's speech patterns, making them unable to hold conversations similar to the user's. Furthermore, there are only a limited number of systems that can hold conversations with third parties on behalf of the user. Furthermore, they lack the ability to use generative AI models to generate responses based on user input and display them to the user. This limits the user experience and makes efficient communication difficult.

[0210] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0211] In this invention, the server includes means for learning a user's utterance patterns, means for generating a clone of the user based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for generating responses using a generative AI model based on user input, and means for displaying the generated responses to the user. This makes it possible to learn a user's utterance patterns, generate a clone that converses similarly to the user, and further generate and display responses based on the user's input using the generative AI model.

[0212] "User utterance patterns" refer to the characteristics and tendencies of a series of utterances and questions made by a user.

[0213] "Means for learning" refers to a method or device for analyzing a user's speech patterns, extracting and storing their characteristics.

[0214] A "clone" refers to a virtual entity that is generated based on the user's speech patterns and can have a conversation similar to that of the user.

[0215] "Means for generating" refers to a method or device for creating clones based on learned speech patterns.

[0216] "Chat" refers to a conversation between a user and a clone via text or voice.

[0217] "Means for updating" refers to a method or device for modifying or adding speech patterns based on new information obtained by the clone through chat with the user.

[0218] A "generative AI model" refers to an artificial intelligence model that generates appropriate responses based on user input.

[0219] "Means for generating a response" refers to a method or device for using a generative AI model to create an appropriate response to a user input.

[0220] "Means for displaying" refers to a method or device for visually or audibly presenting the generated response to the user.

[0221] A system for implementing this invention learns a user's speech patterns, generates clones, and uses a generative AI model to generate and display responses based on the user's input. A specific embodiment of this system is described below.

[0222] System Configuration

[0223] The system mainly consists of the following components:

[0224] 1. User device: A device where the user inputs data, such as a smartphone or computer.

[0225] 2. Server: A central processing unit that learns the user's speech patterns, generates clones, and generates responses using generative AI models.

[0226] 3. Generative AI models: Artificial intelligence models that generate appropriate responses based on user input, such as OpenAI's API.

[0227] Program processing

[0228] User terminal

[0229] The user terminal is responsible for receiving input from the user and sending it to the server. When the user types "What's the weather like today?", the data is sent to the server.

[0230] server

[0231] The server performs the following process:

[0232] 1. Learning speech patterns: Analyzes the user's past speech data, extracts and memorizes its features, and learns the user's speech patterns.

[0233] 2. Clone generation: Based on the learned speech patterns, a clone is generated that can have a similar conversation to the user.

[0234] 3. Use of generative AI models: Based on user input, a generative AI model (e.g., OpenAI's API) is used to generate an appropriate response.

[0235] 4. Displaying the response: The generated response is sent to the user terminal and displayed to the user.

[0236] Specific examples

[0237] When a user asks, "What's the weather like today?", the server uses the generative AI model to generate a response such as "It's sunny," and sends it to the user's device, which then displays the response to the user.

[0238] Prompt Sentence Examples

[0239] An example of a prompt sent to the generative AI model is as follows:

[0240] User: What's the weather like today?

[0241] clone:

[0242] In this way, a system is realized that learns a user's speech patterns, generates clones, and uses a generative AI model to generate and display responses based on the user's input.

[0243] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0244] Step 1:

[0245] The user terminal receives input from the user. Specifically, the user inputs "What's the weather like today?" This input data is sent to the server in text format.

[0246] Step 2:

[0247] The server analyzes the input data received from the user. The input data is analyzed and recorded as data for learning the user's speech patterns. In this step, the server also references the user's past speech data and updates the speech patterns.

[0248] Step 3:

[0249] The server generates a clone based on the learned speech patterns. The clone is a virtual entity that can have a conversation similar to the user. The generated clone reflects the user's speech patterns.

[0250] Step 4:

[0251] The server sends the user's input data to the generative AI model. Specifically, it sends a prompt to the generative AI model (e.g., OpenAI's API). An example of a prompt is "User: What's the weather like today?\nClone: ​​"

[0252] Step 5:

[0253] The generative AI model generates an appropriate response based on the prompt sentence, and the generated response is returned to the server in text format. For example, the response generated might be "It's sunny."

[0254] Step 6:

[0255] The server receives the response returned by the generative AI model and sends it to the user device. The response data is sent to the user device in text format.

[0256] Step 7:

[0257] The user terminal displays the response received from the server to the user. Specifically, the response "It's sunny" is displayed on the user's screen.

[0258] The above processing steps realize a system that learns a user's speech patterns, generates clones, and uses a generative AI model to generate and display responses based on the user's input.

[0259] Example 3

[0260] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0261] In conventional systems, there was no way for the user to take over conversations with people they did not like, which often caused stress for the user. It was also difficult to generate a clone that accurately imitated the user's speech patterns. Furthermore, there was a lack of a way to save and later check the conversation log between the clone and a third party.

[0262] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0263] In this invention, the server includes means for collecting user utterance data, means for preprocessing the collected utterance data, means for training a generative AI model using the preprocessed data, means for generating a clone of the user based on the trained generative AI model, means for using the generated clone to act on behalf of the user in a conversation with a third party, and means for saving a conversation log between the clone and the third party. This allows the user to leave conversations with people the user does not like to the clone, thereby reducing the user's stress. Furthermore, because the clone has similar utterance patterns to the user, a third party may mistakenly believe the clone to be the user. Furthermore, by saving the conversation log, the user can later check the content of the conversation.

[0264] "User" means an individual or organization that uses the system.

[0265] "Utterance data" is text information of conversations and messages that a user has had in the past.

[0266] "Preprocessing" is the process of removing noise from collected speech data and converting it into a format that is easy to analyze.

[0267] A "generative AI model" is a model that uses artificial intelligence technology to learn and imitate user speech patterns.

[0268] A "clone" is a virtual entity that mimics a user's speech patterns based on a generative AI model.

[0269] A "third party" is any person or entity other than the user or clone with whom the conversation takes place.

[0270] A "conversation log" is a record of a conversation between a clone and a third party.

[0271] "Means of collection" refers to methods and techniques for acquiring user utterance data.

[0272] "Preprocessing means" refers to methods and techniques for converting collected speech data into a format that is easy to analyze.

[0273] A "training tool" is a method or technique for training a generative AI model using preprocessed data.

[0274] "Means of generation" refers to methods and technologies for creating clones based on a trained generative AI model.

[0275] "Means of acting as a proxy" refers to methods or techniques for using a clone to converse with a third party.

[0276] "Means of preservation" refers to methods or technologies for recording conversation logs between the clone and a third party so that they can be reviewed later.

[0277] MODE FOR CARRYING OUT THE INVENTION

[0278] The present invention relates to a system that learns a user's speech patterns, generates a clone, and carries out a conversation with a third party on behalf of the user. A specific embodiment of this system will be described below.

[0279] System configuration

[0280] This system consists of three main components: a server, a terminal, and a user. The server collects user speech data, preprocesses it, and trains the generative AI model. The terminal uses the generated clone to converse with a third party and stores the conversation log. The user is an individual or organization that uses the system.

[0281] Hardware and software used

[0282] Server: A server with high-performance computing power is required. For example, a cloud server equipped with a GPU is suitable.

[0283] Generative AI models: Use advanced generative AI models such as OpenAI's GPT-4.

[0284] Data collection tools: Use tools to collect data from chat apps, emails, and social media via APIs, such as the Slack API and Gmail API.

[0285] Data preprocessing tools: Preprocess data using regular expressions or natural language processing libraries (e.g., NLTK or spaCy).

[0286] System Operation

[0287] The server first collects user comment data. For example, it obtains data from Slack or Gmail, which the user uses on a daily basis, via an API. The obtained data is saved as a text file.

[0288] The server then preprocesses the collected speech data, using regular expressions to remove unnecessary HTML tags and special characters, split the text into words, and standardize the case.

[0289] The server uses the preprocessed data to train a generative AI model. The preprocessed data is divided into training data (80%) and test data (20%), and the training data is trained using a GPT-4 model. The server may include means for training the generative AI model based on the updated utterance patterns. Once training is complete, the performance of the model is evaluated using test data to confirm that the accuracy is 90% or higher.

[0290] Based on the trained generative AI model, the server generates a clone of the user, which is then able to make similar statements to the user by applying the user's speech patterns.

[0291] The device uses the generated clone to carry out conversations with third parties. For example, if a user entrusts a conversation with a boss they don't get along with to the clone, and the boss asks, "How's the progress this week?" the clone will respond in real time with, "It's going well. There are no particular problems." After the conversation ends, the device saves the conversation log so the user can check it later.

[0292] Examples of specific examples and prompts

[0293] Examples:

[0294] The user can leave conversations with their boss, whom they don't like, to the clone, which will then naturally converse with the boss based on the user's past utterance data.

[0295] The clone generates answers to questions from its boss that a user would normally give. For example, if the boss asks, "How is things going this week?", the clone might respond, "Everything is going well. No particular issues."

[0296] Example prompt sentence:

[0297] "Generate a clone that mimics conversations with a boss based on the user's past speech data."

[0298] "Create a program that allows the user to delegate conversations with people they don't like to a clone."

[0299] In this way, the user can leave conversations with people the user does not like to the clone. Because the clone has the same speech patterns as the user, a third party may mistakenly recognize the clone as the user. Furthermore, by saving the conversation log, the user can check the content of the conversation later. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[0300] Step 1: Collect user utterance data

[0301] The server collects user comment data. Specifically, it obtains data from chat apps, emails, and social media that users use on a daily basis. For example, it uses the Slack API or Gmail API to obtain messages from the past six months.

[0302] Input: User's chat app, email, and social networking account information

[0303] Output: Text file of acquired speech data

[0304] Step 2: Data Preprocessing

[0305] The server preprocesses the collected utterance data. Specifically, it performs the following processes.

[0306] Noise removal: Remove unnecessary HTML tags and special characters using regular expressions.

[0307] Tokenization: Dividing text data into words and phrases.

[0308] Normalization: unifying case and expanding abbreviations.

[0309] Input: Text file of acquired speech data

[0310] Output: Preprocessed text data

[0311] Step 3: Model training

[0312] The server trains the generative AI model using the preprocessed data. Specifically, it performs the following processes:

[0313] Dataset creation: Split the preprocessed data into training data (80%) and test data (20%).

[0314] Model training: A generative AI model (e.g., GPT-4) is trained on training data.

[0315] Model evaluation: Evaluate the performance of the trained model using test data.

[0316] Input: Preprocessed text data

[0317] Output: A trained generative AI model

[0318] Step 4: Clone Generation

[0319] The server generates clones using a trained generative AI model. Specifically, it performs the following processes:

[0320] Clone initialization: Initialize the clone based on the trained model.

[0321] Applying speech patterns: Applying the user's speech patterns to the clone so that it can speak in the same way as the user.

[0322] Input: A trained generative AI model

[0323] Output: A clone with the user's utterance patterns

[0324] Step 5: Clone conversation

[0325] The terminal uses the clone to act as a proxy for the conversation with the third party. Specifically, it performs the following processes.

[0326] Start conversation: The device launches the clone and starts a conversation with the third party.

[0327] Real-time response: The clone responds to questions and comments from third parties in real time based on the user's speech patterns.

[0328] Saving conversation logs: The device saves conversation logs between the clone and the third party, allowing the user to review them later.

[0329] Input: Clone with user's speech patterns, questions and statements from third parties

[0330] Output: Conversation log between clone and third party

[0331] In this way, users can leave conversations with people they don't like to the clone. Because the clone has the same speech patterns as the user, a third party may mistakenly believe that the clone is the user. Furthermore, by saving the conversation log, users can check the content of the conversation later.

[0332] (Application example 3)

[0333] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0334] Conventional clone generation systems only imitate the user's speech patterns, and it is difficult to achieve the accuracy required to mislead third parties in real conversations. Furthermore, when the clone takes over a conversation with someone or in a situation that the user is not comfortable with, the clone's responses often become unnatural. Furthermore, the application of clones to security checks and authentication processes has not been sufficiently considered.

[0335] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for learning a user's utterance patterns, means for generating a user clone based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for generating the clone's responses using a generative AI model, and means for generating prompt sentences and inputting the clone's responses to the generative AI model. This makes it possible to generate a clone that imitates a user's utterance patterns with high accuracy and has such natural conversations that it can mislead third parties. Furthermore, the application of clones to perform security checks and authentication processes is realized.

[0336] "User speech patterns" refer to the characteristics of the words and speaking style that a user uses on a daily basis.

[0337] "Means for learning" refers to a method or device for analyzing a user's speech patterns and storing their characteristics as data.

[0338] A "clone" refers to an artificial intelligence that imitates a user's speech patterns and converses on their behalf.

[0339] "Means for generating" refers to a method or device for creating a clone based on the learned speech patterns of a user.

[0340] The "means for updating" refers to a method or device for continuously improving the speech patterns of the clone through chat between the generated clone and the user.

[0341] "Generative AI model" refers to a model for generating text using artificial intelligence.

[0342] "Means for generating responses" refers to a method or device for using a generative AI model to generate the clone's utterances.

[0343] A "prompt sentence" refers to an instruction sentence to be input into a generative AI model.

[0344] "Means for input" refers to a method or device for passing a prompt sentence to a generative AI model and generating a response.

[0345] A system for implementing this invention learns a user's speech patterns, generates a clone based on those patterns, and updates the clone's speech patterns through chat between the generated clone and the user. The system further includes means for generating a response for the clone using a generative AI model, generating a prompt sentence, and inputting the clone's response to the generative AI model. The server may include a generation means for using the generative AI model to generate a response for the clone based on the emotion by inputting a specific prompt sentence to the generative AI model, and a means for using the generated response of the clone in a conversation with a third party. The generation means may generate the clone based on the learned speech patterns using a trained generative AI model.

[0346] To learn the user's speech patterns, the server collects the user's past conversation data and analyzes it using natural language processing technology. Specifically, it uses software such as Python and TENSORFLOW® to model the user's speech patterns.

[0347] The server then generates a clone based on the learned speech patterns. The clone can then mimic the user's speech patterns and engage in conversations similar to the user's. The clone is generated using OpenAI's GPT-3 model.

[0348] The generated clones update their speech patterns through chats with the user, allowing them to always reflect the latest user speech patterns. The updated speech patterns are then fed back into the model to improve the accuracy of the clones.

[0349] Furthermore, the server generates a response for the clone using a generative AI model. The server may include a generation means for using the generative AI model to generate a response for the clone according to the emotion by inputting a specific prompt sentence into the generative AI model, and a means for using the generated response for the clone in a conversation with a third party. The generation means may use a trained generative AI model to generate the clone based on the learned utterance pattern. Specifically, a prompt sentence is generated and input into a GPT-3 model to generate a response for the clone. Examples of prompt sentences include the following:

[0350] Example prompt sentence:

[0351] Generate the following conversation based on the user's speech patterns:

[0352] User profile: The user is a man in his 30s who works in the technical field. He speaks politely and logically.

[0353] User Input: Hello, how can I help you today?

[0354] Clone response:

[0355] Using this prompt, the GPT-3 model generates a response that mimics the user's speech pattern, for example, "Hello, do you need technical support today?"

[0356] This system makes it possible to generate clones that can accurately mimic a user's speech patterns and converse so naturally that it can be easily mistaken for a third party. It also enables the application of clones to perform security checks and authentication processes.

[0357] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0358] Step 1:

[0359] The server collects the user's past conversation data. Specifically, it acquires text data such as chat applications and emails used by the user. The input is the user's past conversation data, and the output is text data for analysis.

[0360] Step 2:

[0361] The server analyzes the collected conversation data using natural language processing technology. Specifically, it uses Python and TensorFlow to model the user's speech patterns. The input is the collected text data, and the output is a model that represents the user's speech patterns.

[0362] Step 3:

[0363] The server generates clones based on the learned speech patterns. Specifically, it uses OpenAI's GPT-3 model to create clones that mimic the user's speech patterns. The input is a model that represents the user's speech patterns, and the output is the generated clone.

[0364] Step 4:

[0365] The server updates the clone's speech patterns through chats between the generated clone and the user. Specifically, it collects chat data between the user and the clone and feeds it back into the model. The input is the chat data between the user and the clone, and the output is a model that represents the updated speech patterns.

[0366] Step 5:

[0367] The server generates a response for the clone using a generative AI model. The server may include a generating means for using the generative AI model to generate a response for the clone according to the emotion by inputting a specific prompt sentence into the generative AI model, and a means for using the generated response for the clone in a conversation with a third party. Specifically, the server generates a prompt sentence and inputs it into a GPT-3 model to generate a response for the clone. The input is the prompt sentence, and the output is the response for the clone.

[0368] Step 6:

[0369] The server sends the clone's response to the user or a third party. Specifically, it sends the response via a chat application or email. The input is the clone's response, and the output is the message sent to the user or a third party.

[0370] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0371] "Example 1"

[0372] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system learns a user's speech patterns and generates a clone of the user based on the learned speech patterns. The clone's speech patterns are updated through chats between the generated clone and the user. Furthermore, this system includes an emotion engine that recognizes the user's emotions.

[0373] "Example 2"

[0374] The emotion engine recognizes emotions from the user's words and actions. For example, if a user says, "Today was fun," the emotion engine recognizes from this statement that the user is feeling happy. This recognition result is used to update the clone's speech patterns.

[0375] "Example 3"

[0376] Specifically, if the emotion engine recognizes joy, the clone is updated to make similar statements in situations that make the user feel joy. For example, the clone will say, "Today was fun." This allows the clone to make statements that reflect the user's emotions.

[0377] "Example 4"

[0378] The emotion engine is also used by the clone to adjust its conversation with third parties. For example, if the emotion engine recognizes that the user is feeling angry, the clone will use that information to appropriately adjust its conversation with third parties. Specifically, in situations where the user feels angry, the clone will make statements that express the user's anger.

[0379] The processing flow of each embodiment will be described below.

[0380] "Example 1"

[0381] Step 1: The system initiates a chat with the user through a messaging app.

[0382] Step 2: The system learns the user's speech patterns through chat.

[0383] Step 3: Based on the learned speech patterns, the system generates a clone of the user.

[0384] Step 4: Through chat between the generated clone and the user, the system updates the clone's speech patterns.

[0385] Step 5: The system activates the emotion engine to recognize the user's emotions.

[0386] "Example 2"

[0387] Step 1: The emotion engine recognizes emotions from the user's statements and actions.

[0388] Step 2: The recognized emotions are used to update the clone's speech patterns.

[0389] "Example 3"

[0390] Step 1: When the emotion engine recognizes a particular emotion, the clone's speech pattern is updated to reflect that emotion.

[0391] Step 2: Based on the updated speech patterns, the clone makes speech that expresses similar emotions to the user.

[0392] "Example 4"

[0393] Step 1: The emotion engine recognizes the user's emotion.

[0394] Step 2: Based on the perceived emotion, the clone adjusts the conversation with the third party.

[0395] Step 3: Specifically, the clone makes statements that reflect the emotions the user might be feeling.

[0396] Example 1

[0397] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0398] Conventional chat systems can learn a user's speech patterns and generate a user clone, but they lack the ability to recognize the user's emotions and generate responses accordingly. This makes it difficult to communicate in a way that is sensitive to the user's emotions, limiting the user experience.

[0399] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for collecting user utterance data, means for preprocessing the collected utterance data, means for learning utterance patterns using the preprocessed utterance data, means for generating a user clone based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for recognizing emotions from the user's utterances, and means for generating responses according to the recognized emotions. This makes it possible to learn the user's utterance patterns and generate responses according to the emotions.

[0400] The "means of collecting user comment data" refers to a function that saves comments made by users through the messenger app as a log and sends them to a server.

[0401] The "means for preprocessing collected utterance data" is a function that converts collected utterance data into a format that is easy to analyze by processing the data such as tokenizing, removing stop words, and stemming.

[0402] The "means for learning utterance patterns using preprocessed utterance data" is a function that uses a machine learning algorithm based on preprocessed data to analyze user utterance patterns and train a model.

[0403] The "means for generating a clone of a user based on learned speech patterns" is a function for creating a virtual agent that mimics the speech patterns of a user based on a learned model.

[0404] The "means for updating the clone's speech patterns through chat between the generated clone and the user" is a function that continuously learns and updates the clone's speech patterns using newly collected speech data through chat between the clone and the user.

[0405] The "means for recognizing emotions from user utterances" is a function that analyzes the content of a user's utterances and identifies the user's emotions using an emotion engine.

[0406] The "means for generating a response according to the recognized emotion" is a function for generating an appropriate response based on the user's emotion identified by the emotion engine.

[0407] This invention is a system that learns a user's speech patterns and generates a clone of the user based on the learning results. Furthermore, the generated clone updates its speech patterns through chats with the user, recognizes the user's emotions using an emotion engine, and generates responses accordingly.

[0408] The server uses a messenger app to collect user speech data. All speech made by users within the app is saved as a log and sent to the server. The collected speech data is preprocessed using natural language processing libraries such as Python's NLTK and spaCy. Preprocessing includes tokenizing the text data, removing stop words, and stemming.

[0409] The preprocessed speech data is used to learn speech patterns using machine learning frameworks such as TensorFlow and PyTorch. Specifically, a recurrent neural network (RNN) or transformer model is constructed, the user's speech patterns are analyzed, and the model is trained. A clone of the user is generated based on the learned model. The clone is a virtual agent that mimics the user's speech patterns.

[0410] The device then starts a chat between the user and the clone. When the user speaks through a messaging app, the clone responds accordingly. For example, if the user says "Good morning," the clone responds with "Good morning!"

[0411] Newly collected speech data from chats with the clone is sent to the server and used to update the clone's speech patterns, allowing the clone to reflect the user's latest speech patterns.

[0412] Furthermore, the device uses an emotion engine to recognize emotions from the user's speech. The emotion engine uses Microsoft Azure's (registered trademark) Text Analytics API or IBM Watson's (registered trademark) Tone Analyzer, among others. Once the emotion engine recognizes the user's emotion, it generates an appropriate response according to that emotion. For example, if the user says, "I'm tired today," the emotion engine will recognize the user's sense of fatigue and respond, "Thank you for your hard work. Please take a good rest today."

[0413] Example prompt sentence:

[0414] User: Good morning

[0415] Clone: ​​Good morning!

[0416] User: I'm tired today

[0417] Clone: ​​Thank you for your hard work. Please get some rest today.

[0418] In this way, the server, terminal, and user work together to learn the user's speech patterns, generate clones, and respond according to their emotions.

[0419] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0420] Step 1: Collect user utterance data

[0421] The server collects user speech data through the messenger app. When a user speaks within the app, the speech is saved as a log and sent to the server. The input is the user's speech text, and the output is the speech log saved on the server. Specifically, when a user says "Good morning," the text data is sent to the server.

[0422] Step 2: Preprocessing the utterance data

[0423] The server preprocesses the collected utterance data. The input is the collected utterance data, and the output is preprocessed text data. Specifically, it uses Python's NLTK and spaCy to tokenize the text data, remove stop words, and perform stemming. For example, the utterance "Good morning" is tokenized and unnecessary characters are removed.

[0424] Step 3: Learning speech patterns

[0425] The server uses preprocessed utterance data to learn utterance patterns. The input is preprocessed text data, and the output is a trained model. Specifically, a recurrent neural network (RNN) or transformer model is built using TensorFlow or PyTorch, and the user's utterance patterns are analyzed to train the model. For example, the utterance pattern "Good morning" is learned by the model.

[0426] Step 4: Creating a clone

[0427] The server generates a user clone based on the trained model. The input is the trained model, and the output is the user clone. The clone is a virtual agent that imitates the user's speech patterns. For example, a clone that says "Good morning" is generated.

[0428] Step 5: Chat with the clone

[0429] The device starts a chat between the generated clone and the user. The input is the user's speech, and the output is the clone's response. When the user speaks through a messenger app, the clone responds accordingly. For example, if the user says "Good morning," the clone responds "Good morning!"

[0430] Step 6: Update the clone's speech patterns

[0431] The server updates the clone's speech patterns using newly collected speech data through chats with the clone. The input is the newly collected speech data, and the output is the updated model. This allows the clone to reflect the user's latest speech patterns. For example, when a user says, "I'm tired today," that data is sent to the server, and the clone's speech patterns are updated.

[0432] Step 7: Emotion recognition and response generation by the emotion engine

[0433] The device uses an emotion engine to recognize emotions from what the user says. The input is what the user says, and the output is the recognized emotion and an appropriate response. Specifically, it uses Microsoft Azure's Text Analytics API and IBM Watson's Tone Analyzer. Once the emotion engine recognizes the user's emotion, it generates an appropriate response based on that emotion. For example, if the user says, "I'm tired today," the emotion engine will recognize the user's sense of fatigue and respond, "Thank you for your hard work. Please take it easy and rest today."

[0434] (Application example 1)

[0435] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0436] Conventional user interfaces and content recommendation systems have had difficulty fully understanding a user's speech patterns and emotions and providing individually optimized content. Furthermore, there were no systems that could converse with a third party on the user's behalf, which placed a heavy burden on the user.

[0437] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for learning a user's speech patterns, means for generating a user clone based on the learned speech patterns, means for updating the clone's speech patterns through chat between the generated clone and the user, means for recognizing the user's emotions, and means for recommending appropriate content based on the recognized emotions. This makes it possible to understand the user's speech patterns and emotions and provide individually optimized content. It is also possible to have a conversation with a third party on behalf of the user.

[0438] "User speech patterns" refers to the tendencies and characteristics of speech made by users through messenger apps and chat systems.

[0439] "Means of learning" refers to algorithms and systems that analyze users' speech patterns and accumulate their characteristics as data.

[0440] A "clone" is a virtual entity that is generated based on a user's speech patterns and imitates the same speech and behavior as the user.

[0441] "Means for updating the clone's speech patterns through chat" refers to a function that enables the clone to learn new speech patterns through dialogue with the user and to modify and update its own speech patterns.

[0442] "Means for recognizing emotions" refers to algorithms or systems that analyze emotions from a user's words and actions and identify their emotional state.

[0443] "Means for recommending appropriate content" refers to a function for selecting and providing the most suitable content (movies, music, articles, etc.) to a user based on the user's recognized emotions.

[0444] As an embodiment of the present invention, a system is constructed that learns user speech patterns, generates clones, recognizes emotions, and recommends appropriate content. A specific embodiment of this system is shown below.

[0445] System Configuration

[0446] The system consists of the following main components:

[0447] 1. User device: A device such as a smartphone or computer that users use to input their comments.

[0448] 2. Server: A central processing unit for learning user speech patterns, generating clones, recognizing emotions, and recommending content.

[0449] 3. Emotion Recognition Engine: Software that uses emotion analysis tools such as TextBlob to recognize emotions from user speech.

[0450] 4. Generative AI model: An algorithm that uses OpenAI's API to generate clone responses based on the user's speech patterns.

[0451] Program processing

[0452] The server performs the following process.

[0453] 1. A means of learning user speech patterns: Analyzes speech sent from the user's device and stores the speech patterns in a database. This allows us to understand the user's speech tendencies.

[0454] 2. Method for generating clones: Based on the learned speech patterns, a generative AI model is used to generate a clone of the user, which can make similar speech to the user.

[0455] 3. A method for updating the clone's speech patterns: Through chats between the clone and the user, new speech patterns are learned and the clone's speech patterns are updated.

[0456] 4. Means of Emotion Recognition: An emotion recognition engine is used to analyze emotions from the user's speech and identify their emotional state.

[0457] 5. Means of recommending appropriate content: Based on the recognized emotions, the most suitable content (movies, music, articles, etc.) for the user is selected and provided to the user's device.

[0458] Hardware and software used

[0459] Hardware: smartphones, PCs, servers

[0460] Software: Python, OpenAI API, TextBlob

[0461] Specific examples

[0462] If the user says "I'm so happy today!", the following happens:

[0463] 1. A message is sent from the user's device to the server.

[0464] 2. The server learns speech patterns and stores them in a database.

[0465] 3. Using a generative AI model, the clone responds, "That's great!"

[0466] 4. The emotion recognition engine analyzes the user's speech and recognizes positive emotions.

[0467] 5. The server recommends content based on positive emotions, such as "Here is a happy movie for you!"

[0468] Prompt Sentence Examples

[0469] User says: I'm so happy today!

[0470] AI Clone responds:

[0471] In this way, a system is realized that understands the user's speech patterns and emotions and provides individually optimized content.

[0472] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0473] Step 1:

[0474] A message is sent from the user terminal to the server.

[0475] Input: User utterance (e.g., "I'm so happy today!")

[0476] Processing: The user terminal sends the utterance as text data to the server.

[0477] Output: Speech data arrives at the server.

[0478] Step 2:

[0479] The server learns speech patterns and stores them in a database.

[0480] Input: User's speech data

[0481] Processing: The server analyzes the speech data and extracts speech patterns. The extracted patterns are stored in a database.

[0482] Output: Updated utterance pattern database

[0483] Step 3:

[0484] Using a generative AI model, the clone generates a response.

[0485] Input: User utterance data, utterance pattern database

[0486] Processing: The server uses the OpenAI API to generate a clone's response based on the user's utterance. The server inputs the prompt sentence into the generative AI model and obtains the response.

[0487] Output: The clone's response (e.g., "That's great!")

[0488] Step 4:

[0489] The clone's response is sent to the user terminal.

[0490] Input: Clone response

[0491] Processing: The server sends the generated clone response to the user terminal.

[0492] Output: The clone's response displayed on the user's terminal

[0493] Step 5:

[0494] The emotion recognition engine analyzes the user's speech and recognizes their emotions.

[0495] Input: User's speech data

[0496] Processing: The server uses TextBlob to parse the sentiment from the user's speech and identify positive, negative, and neutral sentiment states.

[0497] Output: Recognized emotion data (e.g., positive)

[0498] Step 6:

[0499] The server recommends appropriate content based on the recognized emotions.

[0500] Input: Recognized emotion data

[0501] Processing: The server selects the most suitable content for the user based on the emotion data. For example, if the emotion is positive, it will select content such as "Here is a happy movie for you!"

[0502] Output: Recommended content

[0503] Step 7:

[0504] The recommended content is transmitted to the user terminal.

[0505] Input: Recommended content

[0506] Processing: The server sends the recommended content to the user terminal.

[0507] Output: Recommended content displayed on the user's device

[0508] In this way, a system is realized that understands the user's speech patterns and emotions and provides individually optimized content.

[0509] Example 2

[0510] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0511] Conventional systems were able to learn the user's speech patterns and generate clones, but they were unable to recognize the user's emotions and dynamically update the clone's speech patterns based on those emotions. As a result, conversations with the user were one-sided, making it difficult to have natural dialogue. Furthermore, the inability to respond to the user's emotions led to a poor user experience.

[0512] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for learning a user's utterance patterns, means for generating a clone of the user based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for recognizing emotions from the user's utterances and actions, and means for updating the clone's utterance patterns based on the recognized emotions. This enables natural conversation according to the user's emotions.

[0513] The "means for learning user speech patterns" is a function that analyzes speech made by a user, extracts and records their characteristics and tendencies.

[0514] The "means for generating a user clone" is a function that creates a virtual entity that makes statements similar to those of the user based on the learned speech patterns of the user.

[0515] The "means for updating the clone's speech patterns" is a function that improves the content of the clone's speech by reflecting new information and speech patterns that the generated clone has acquired through chat with the user.

[0516] "Means for recognizing emotions from user's statements and actions" is a function that analyzes the statements and actions made by the user and identifies the emotions behind them.

[0517] "Means for updating the clone's speech patterns based on recognized emotions" is a function that adjusts the content and tone of the clone's speech based on the results of emotion recognition, thereby realizing more natural and emotionally appropriate dialogue.

[0518] MODE FOR CARRYING OUT THE INVENTION

[0519] This invention is a system that learns a user's speech patterns, generates a clone, and updates the clone's speech patterns through dialogue with the user. It can also recognize emotions from the user's speech and behavior, and dynamically update the clone's speech patterns based on those emotions.

[0520] Hardware and software used

[0521] Hardware: Servers, terminals (PCs, smartphones)

[0522] Software: Generative AI models (e.g., GPT-4), emotion recognition engines (e.g., IBM Watson Tone Analyzer)

[0523] Specific operation of the system

[0524] 1. Accepting user input

[0525] The device receives the user's comments and questions in text format.

[0526] Example: A user types "What's the weather like today?" into a chat app on their smartphone.

[0527] The terminal transmits this input to the server in real time.

[0528] 2. Answer Generation by Cloning

[0529] The server receives user input from the device and sends it to the generative AI model.

[0530] Example: The server sends the user input "What's the weather like today?" as a prompt to a generative AI model (e.g., GPT-4).

[0531] A generative AI model generates appropriate answers based on user input.

[0532] Example: The generative AI model generates the answer "It's sunny."

[0533] The server sends the generated response to the terminal.

[0534] The terminal displays the answer to the user.

[0535] Example: The device displays "It's sunny" in a chat app.

[0536] 3. Emotional Recognition

[0537] The server sends the user's utterances to the emotion recognition engine.

[0538] Example: If a user says, "I had fun today," the server sends this statement to an emotion recognition engine (e.g., IBM Watson Tone Analyzer).

[0539] The emotion recognition engine analyzes emotions from user speech.

[0540] Example: An emotion recognition engine recognizes "joy" from the statement "Today was fun."

[0541] The server receives the results of the emotion recognition engine.

[0542] Example: The server receives the recognition result "joy" from the emotion recognition engine.

[0543] 4. Updating the clone's speech patterns

[0544] The server updates the clone's speech patterns based on the results of the emotion recognition engine.

[0545] Example: Based on the emotion recognition result of "joy," the server updates the clone's speech pattern in a positive direction.

[0546] Set the clone to say something positive in the next conversation, such as "Today was fun!"

[0547] Prompt Sentence Examples

[0548] "When a user asks, 'What's the weather like today?' how do you respond?"

[0549] "When a user says, 'I had fun today,' what emotion do you recognize?"

[0550] In this way, the system can have a natural conversation with the user and respond according to the user's emotions.

[0551] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0552] Step 1:

[0553] Accepting user input

[0554] The device receives the user's comments and questions in text format.

[0555] Input: A user types "What's the weather like today?" into a chat app on their smartphone.

[0556] Specific operation: The device sends this input to the server in real time.

[0557] Output: User's text input sent to the server.

[0558] Step 2:

[0559] Answer generation by cloning

[0560] The server receives user input from the device and sends it to the generative AI model.

[0561] Input: The server sends the user input "What's the weather like today?" as a prompt to the generative AI model.

[0562] What happens: A generative AI model (e.g., GPT-4) generates an appropriate answer based on user input.

[0563] Output: The answer generated by the generative AI model: "It's sunny."

[0564] Step 3:

[0565] Submitting and viewing answers

[0566] The server sends the generated response to the terminal.

[0567] Input: The answer from the generative AI model: "It's sunny."

[0568] Specific operation: The server receives the answer from the generative AI model and sends it to the device.

[0569] The terminal displays the answer to the user.

[0570] Output: The device displays "It's sunny" in the chat app.

[0571] Step 4:

[0572] Emotion recognition

[0573] The server sends the user's utterances to the emotion recognition engine.

[0574] Input: Text data when a user says "Today was fun."

[0575] Specific operation: The server sends this utterance to an emotion recognition engine (e.g., IBM Watson Tone Analyzer).

[0576] The emotion recognition engine analyzes emotions from user speech.

[0577] Output: The emotion "joy" recognized by the emotion recognition engine.

[0578] Step 5:

[0579] Receiving emotion recognition results

[0580] The server receives the results of the emotion recognition engine.

[0581] Input: The recognition result "joy" from the emotion recognition engine.

[0582] What it does: The server receives the results from the emotion recognition engine and uses them to update the clone's speech patterns.

[0583] Output: Data for updating the clone's utterance patterns.

[0584] Step 6:

[0585] Updated clone speech patterns

[0586] The server updates the clone's speech patterns based on the results of the emotion recognition engine.

[0587] Input: Emotion recognition result "Joy".

[0588] Specific operation: Based on the emotion recognition result of "joy," the server updates the clone's speech pattern in a positive direction.

[0589] Output: Updated clone utterance patterns.

[0590] In this way, the system can have a natural conversation with the user and respond according to the user's emotions.

[0591] (Application example 2)

[0592] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0593] Conventional systems that learn user speech patterns and generate clones were unable to recognize the user's emotions and respond appropriately based on them. This made it difficult to provide appropriate service through dialogue with customers, especially in physical stores. Furthermore, there was a lack of a way to recognize customer emotions in real time and provide feedback to store staff, making it difficult to improve customer satisfaction.

[0594] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0595] In this invention, the server includes means for learning a user's speech patterns, means for generating a clone of the user based on the learned speech patterns, means for updating the clone's speech patterns through chat between the generated clone and the user, means for recognizing the user's emotions, means for updating the clone's speech patterns based on the recognized emotions, and means for recognizing customer emotions in real time and supporting appropriate responses, thereby making it possible to recognize customer emotions in real time and provide appropriate feedback.

[0596] "Means for learning user speech patterns" refers to technology for analyzing the characteristics and trends of user speech and accumulating the data.

[0597] "Means for generating a user clone based on learned speech patterns" is a technology that creates a virtual entity (clone) that makes similar statements to the user based on the accumulated speech patterns of the user.

[0598] "Means for updating the clone's speech patterns through chat between the generated clone and the user" refers to a technology that continuously improves and updates the clone's speech patterns through dialogue between the clone and the user.

[0599] "Means for recognizing user emotions" refers to technology for analyzing and identifying emotions from a user's statements and actions.

[0600] "Means for updating the clone's speech patterns based on the recognized emotions" refers to a technology that appropriately adjusts and updates the clone's speech patterns based on the results of analyzing the user's emotions.

[0601] "Means to recognize customer emotions in real time and support appropriate responses" refers to technology that analyzes customer comments and facial expressions in real time, and instructs and supports store staff on appropriate responses based on those emotions.

[0602] The system for implementing this invention has the functions of learning a user's speech patterns, generating a clone, and recognizing the user's emotions to update the clone's speech patterns. A specific embodiment of this system will be described below.

[0603] First, the server collects and analyzes user utterances to learn their speech patterns. This is done by using natural language processing technology to extract the characteristics and trends of the utterances. Examples of software used include NLTK and spaCy, which are natural language processing libraries for Python.

[0604] Next, based on the learned speech patterns, the server generates a clone of the user. This clone is a virtual entity that speaks in a similar way to the user and is created using a generative AI model, specifically, OpenAI's GPT-3.

[0605] The generated clone updates its speech patterns through chats with the user. The server collects data on the clone's interactions with the user and continuously improves the clone's speech patterns. This process is performed using a machine learning algorithm.

[0606] Furthermore, the server analyzes the user's speech and behavior to recognize the user's emotions. For emotion recognition, it uses Transformers emotion analysis models. For example, it uses the Hugging Face Transformers library to identify emotions from the user's speech.

[0607] Based on the recognized emotion, the server updates the clone's speech patterns, allowing the clone to make appropriate speech according to the user's emotion.

[0608] In a physical store, when a salesperson wearing smart glasses talks to a customer, the glasses recognize the customer's emotions in real time and provide feedback to the salesperson. The smart glasses are equipped with a camera that detects the customer's face using OpenCV and analyzes their emotions using a Transformers emotion recognition model.

[0609] For example, if a customer says, "I really like this product!", the server recognizes that the customer is happy and notifies the store clerk, "The customer is happy. Please suggest additional products."

[0610] An example of a prompt to input to a generative AI model is as follows:

[0611] What customers say: "I absolutely love this product!"

[0612] Emotion recognition result: "POSITIVE"

[0613] Feedback: "Customers are happy. Please suggest additional products."

[0614] In this way, the system can recognize customer emotions in real time and provide appropriate feedback.

[0615] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0616] Step 1:

[0617] The server collects user utterances. It receives the text data uttered by the user as input and uses natural language processing technology to analyze the characteristics and trends of the utterances. Specifically, it uses Python natural language processing libraries NLTK and spaCy to tokenize the utterances, tag parts of speech, and perform grammatical analysis. The analysis results are stored in a database as the user's utterance patterns.

[0618] Step 2:

[0619] The server generates a clone of the user based on the learned speech patterns. It uses the speech patterns collected and analyzed in step 1 as input. It uses a generative AI model (e.g., OpenAI's GPT-3) to create a virtual entity (clone) that speaks similarly to the user. The generated clone mimics the user's speech patterns.

[0620] Step 3:

[0621] The server updates the clone's speech patterns through chats between the generated clone and the user. It collects interaction data between the clone and the user as input. It analyzes the interaction data and applies a machine learning algorithm to continuously improve and update the clone's speech patterns. The updated speech patterns are reflected in the clone's speech model.

[0622] Step 4:

[0623] The server recognizes the user's emotions. It uses the user's speech and behavior data as input. For emotion recognition, it uses a Transformers emotion analysis model (for example, the Hugging Face Transformers library). It identifies emotions from the user's speech and saves the results as emotion data.

[0624] Step 5:

[0625] The server updates the clone's speech patterns based on the recognized emotions. It uses the emotion data recognized in step 4 as input. Based on the emotion data, it adjusts and updates the clone's speech patterns appropriately. This allows the clone to make appropriate speech according to the user's emotions.

[0626] Step 6:

[0627] The device (smart glasses) recognizes the customer's emotions in real time and supports appropriate responses. It uses the customer's facial image captured by the smart glasses' camera as input. It detects the customer's face using OpenCV and analyzes their emotions using a Transformers emotion recognition model. The analysis results are then sent to the store clerk as feedback.

[0628] Step 7:

[0629] The terminal (smart glasses) displays the feedback to the store clerk. It uses the emotion recognition results analyzed in step 6 as input. For example, if the customer says, "I really like this product!", the server recognizes that the customer is happy and notifies the store clerk, "The customer is happy. Please suggest additional products." The notification is displayed on the smart glasses display.

[0630] Example 3

[0631] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0632] Conventional systems were able to learn a user's speech patterns and generate a clone, but they had difficulty generating speech that reflected the user's emotions or appropriately adjusting conversations with third parties. Furthermore, they lacked a mechanism for updating the model based on user feedback, making it difficult to improve the accuracy of the clone's speech. This resulted in the clone being unable to converse naturally on behalf of the user, resulting in a poor user experience.

[0633] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0634] In this invention, the server includes means for learning a user's utterance patterns, means for generating a user clone based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for collecting user utterance data, means for analyzing emotions based on the collected utterance data, means for generating utterances for the clone based on the analysis results, means for using the utterances of the generated clone in conversations with third parties, and means for updating the model based on feedback. This makes it possible to generate natural utterances that reflect the user's emotions and appropriately adjust conversations with third parties. Furthermore, by updating the model based on user feedback, the accuracy of the clone's utterances can be improved.

[0635] "Means for learning user utterance patterns" refers to technology or devices that analyze past utterances made by users and extract their characteristics and trends.

[0636] "Means for generating a user clone" refers to technology or devices that create a virtual entity that makes statements similar to those of the user, based on the learned speech patterns of the user.

[0637] "Means for updating the clone's speech patterns" refers to technology or devices that allow the generated clone to learn new speech patterns through chats with users and improve and update the content of its speech.

[0638] "Means for collecting user utterance data" refers to technology or devices for recording voice or text utterances made by users and saving them as data.

[0639] "Means for analyzing emotions" refers to technology or devices for identifying and analyzing emotions from collected user speech data.

[0640] "Means for generating clone utterances" refers to technology or devices that generate text or audio for the clone to make appropriate utterances based on the results of emotion analysis.

[0641] "Means used for conversation with a third party" refers to the technology or device used to apply the utterances of the generated clone to a conversation with a third party and actually carry out the conversation.

[0642] "Means for updating the model based on feedback" refers to technologies and devices that retrain the generative AI model based on user evaluations and feedback, improving the accuracy of the clone's speech.

[0643] The present invention relates to a system that learns a user's speech patterns, generates a clone that reflects the user's emotions, and has a conversation with a third party. A specific embodiment of this system will be described below.

[0644] First, users generate utterance data through everyday conversations and text messages. The device collects this utterance data in real time and sends it to a server. Specifically, devices such as smartphones and PCs record and record the user's voice and text, and upload them to the server.

[0645] Next, the server inputs the received utterance data into the emotion engine. The emotion engine uses natural language processing technology to analyze the emotion from the user's utterance. Specifically, it uses a general emotion analysis tool. For example, if a user says, "Today was fun," the emotion engine recognizes "joy."

[0646] The server then inputs a prompt sentence into the generative AI model based on the analysis results of the emotion engine. The generative AI model generates the clone's speech that reflects the user's speech patterns and emotions. Specifically, a general generative AI model is used. For example, the prompt sentence "User's speech: Today was fun. Emotion: Joy" is input, and the clone's speech is generated as "Today was really fun."

[0647] The generated clone's speech is used in conversations with third parties. The device outputs the clone's speech as voice using speech synthesis technology. Specifically, a common speech synthesis tool is used. For example, the clone might say, "I had a really good time today," and the device would play that speech.

[0648] Furthermore, users provide feedback on the clone's comments and the results of the conversation. The server updates the generative AI model based on this feedback, improving the accuracy of the clone's comments. Specifically, if a user evaluates a comment as "appropriate," the data is added to the model's training data.

[0649] As a concrete example, consider the case where a user says "Today was fun" in a conversation with a friend. The device collects this utterance data and sends it to the server. The server uses an emotion engine to analyze "joy" and inputs the prompt text "User utterance: Today was fun. Emotion: Joy" into the generative AI model. The generative AI model generates the clone's utterance, "Today was really fun," and the device plays back that audio.

[0650] Example prompt sentence:

[0651] User says: "Today was fun"

[0652] Emotion engine analysis result: "Joy"

[0653] Clone says: "I had a really good time today."

[0654] In this way, the server analyzes the user's emotions and speech patterns and adjusts the clone to make appropriate speech. This makes it possible to generate natural speech that reflects the user's emotions and appropriately adjust conversations with third parties. In addition, by updating the model based on user feedback, the accuracy of the clone's speech can be improved. The flow of the identification process in Example 3 will be described with reference to FIG. 21.

[0655] Step 1:

[0656] Users generate utterance data through everyday conversations and text messages. Devices collect this utterance data in real time and send it to a server. Specifically, devices such as smartphones and PCs record and record users' voices and text, and upload them to a server. The input is the user's utterance data, and the output is the utterance data sent to the server.

[0657] Step 2:

[0658] The server inputs the received utterance data into the emotion engine. The emotion engine uses natural language processing technology to analyze emotions from the user's utterances. Specifically, it uses a general emotion analysis tool. For example, if a user utters, "I had fun today," the emotion engine recognizes "joy." The input is the utterance data, and the output is analyzed emotion data.

[0659] Step 3:

[0660] The server inputs a prompt sentence into the generative AI model based on the analysis results of the emotion engine. The generative AI model generates the clone's utterances that reflect the user's utterance patterns and emotions. Specifically, a general generative AI model is used. For example, the prompt sentence is input as "User's utterance: Today was fun. Emotion: Joy," and the clone's utterance is generated as "Today was really fun." The input is the prompt sentence, and the output is the clone's utterance.

[0661] Step 4:

[0662] The generated clone's utterances are used in conversations with third parties. The device outputs the clone's utterances as voice using speech synthesis technology. Specifically, a common speech synthesis tool is used. For example, the clone may say, "I had a really good time today," and the device plays back the voice. The input is the clone's utterance, and the output is the utterance played back as voice.

[0663] Step 5:

[0664] The user provides feedback on the clone's utterances and the results of the conversation. The server updates the generative AI model based on this feedback, improving the accuracy of the clone's utterances. Specifically, if the user evaluates a utterance as "appropriate," the data is added to the model's training data. The input is the user's feedback, and the output is an updated generative AI model.

[0665] (Application example 3)

[0666] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0667] Conventional systems that learn a user's speech patterns have difficulty responding appropriately when the user faces unpleasant or dangerous situations. Furthermore, they lack the ability to recognize the user's emotions and adjust speech based on those emotions, which can lead to inappropriate responses when conversing with a third party on the user's behalf. This can potentially compromise the user's safety and comfort.

[0668] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0669] In this invention, the server includes means for learning a user's speech patterns, means for generating a clone of the user based on the learned speech patterns, means for updating the clone's speech patterns through chat between the generated clone and the user, means for recognizing the user's emotions, means for adjusting the clone's speech based on the recognized emotions, and means for the clone to take an appropriate response when the user faces an unpleasant or dangerous situation. This enables an appropriate response that reflects the user's emotions, thereby improving the user's safety and comfort.

[0670] "User speech patterns" refer to the words, phrases, and speaking style characteristics that a user uses on a daily basis.

[0671] A "clone" refers to an artificial intelligence that learns a user's speech patterns and converses on their behalf.

[0672] "Chat" refers to text-based interactive communication.

[0673] "Means for recognizing emotions" refers to technology that analyzes emotions from a user's statements and actions and identifies specific emotions.

[0674] "Means to tailor speech" refers to technology that alters the content and tone of a clone's speech based on perceived emotions.

[0675] An "unpleasant situation" refers to a situation in which the user feels psychologically uncomfortable.

[0676] A "dangerous situation" refers to a situation in which the user feels physically or psychologically unsafe.

[0677] "Appropriate response" refers to actions and statements necessary to maintain the safety and comfort of users.

[0678] The system for implementing this invention has the functions of learning a user's speech patterns, generating a clone of the user, and updating the clone's speech patterns through chat between the user and the generated clone. It also has the functions of recognizing the user's emotions, adjusting the clone's speech based on the recognized emotions, and allowing the clone to respond appropriately when the user faces an unpleasant or dangerous situation.

[0679] System Configuration

[0680] The system consists of the following major components:

[0681] 1. User terminal: A device such as a smartphone or tablet used by the user. It provides an interface for inputting user comments and chatting with the clone.

[0682] 2. Server: Learns the user's speech patterns, generates clones, recognizes emotions, and processes the clones' speech to adjust them.

[0683] 3. Emotion recognition engine: Software that analyzes emotions from a user's words and actions and identifies specific emotions.

[0684] 4. Clone generation engine: Software that generates clones based on the user's speech patterns and updates the clones' speech patterns.

[0685] Program processing

[0686] The server receives the speech data sent from the user terminal and analyzes the user's emotions using an emotion recognition engine. Based on the analysis results, the clone generation engine adjusts the clone's speech and takes appropriate action on behalf of the user.

[0687] Specifically, if a user says, "I'm having fun today," the emotion recognition engine will recognize "joy," and the clone generation engine will respond with, "The user is feeling happy." In this way, it is possible to respond appropriately to the user's emotions.

[0688] Hardware and software used

[0689] Hardware: smartphones, tablets, servers

[0690] Software: Emotion recognition engine, clone generation engine

[0691] Specific examples

[0692] For example, if the user inputs "Today is fun," the system operates as follows:

[0693] 1. The user terminal sends the speech data to the server.

[0694] 2. The server uses an emotion recognition engine to recognize "joy."

[0695] 3. The clone generation engine responds, "The user finds it fun."

[0696] Prompt Sentence Examples

[0697] When a user says "I'm having fun today," generate a program in which the emotion engine recognizes "joy" and the clone responds "The user is feeling happy."

[0698] In this way, it is possible to respond appropriately to the user's feelings, thereby improving the safety and comfort of the user.

[0699] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0700] Step 1:

[0701] The user terminal inputs speech data.

[0702] A user uses a smartphone or tablet to input a statement. For example, they might type, "Today is fun." The input data is sent to the server in text format.

[0703] Step 2:

[0704] The server receives the speech data.

[0705] The server receives the utterance data sent from the user terminal and temporarily stores the received data for use in the next processing step.

[0706] Step 3:

[0707] The server analyzes the speech data using an emotion recognition engine.

[0708] The server inputs the received utterance data into an emotion recognition engine. The emotion recognition engine analyzes the text data and identifies the user's emotion. For example, it recognizes "joy" from an utterance such as "Today is fun." The analysis results are output as emotion data.

[0709] Step 4:

[0710] The server generates clone utterances using a clone generation engine.

[0711] The server inputs the emotion data output from the emotion recognition engine into the clone generation engine. The clone generation engine generates appropriate utterances based on the user's utterance patterns and emotion data. For example, if "joy" is recognized, the engine generates the utterance "The user is feeling happy." The generated utterance is output as text data.

[0712] Step 5:

[0713] The server transmits the generated utterance data to the user terminal.

[0714] The server transmits the utterance data output from the clone generation engine to the user terminal, where it is displayed.

[0715] Step 6:

[0716] The user terminal displays the generated utterance data.

[0717] The user terminal receives the utterance data sent from the server and displays it on the screen. For example, a utterance such as "The user is having fun" is displayed. This allows the user to check the clone's response.

[0718] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0719] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0720] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0721] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing is performed by the smart device 14.

[0722] This may be done.

[0723] [Second embodiment]

[0724] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0725] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0726] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0727] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0728] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0729] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0730] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0731] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0732] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0733] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0734] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0735] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0736] "Example 1"

[0737] One embodiment of the present invention is a system in which a user makes statements through a messenger app. This system learns the user's speech patterns and generates a clone of the user based on the learning results. The generated clone updates its speech patterns through chats with the user. Specifically, if the user says "Good morning," the clone also learns to say "Good morning."

[0738] "Example 2"

[0739] Furthermore, the clone can have a conversation similar to that of the user. For example, if the user asks, "What's the weather like today?", the clone will learn to respond, "It's sunny."

[0740] "Example 3"

[0741] The clone can also converse with third parties on behalf of the user. Specifically, the user can leave conversations with people the clone does not like to talk to. Because the clone has the same speech patterns as the user, third parties may mistake the clone for the user.

[0742] The processing flow of each embodiment will be described below.

[0743] "Example 1"

[0744] Step 1: The user makes a comment through the messenger app.

[0745] Step 2: The system learns the user's speech patterns.

[0746] Step 3: Generate a clone of the user based on the learning results.

[0747] Step 4: The generated clone updates its speech patterns through chat with the user. Specifically, if the user says "Good morning," the clone also learns to say "Good morning."

[0748] "Example 2"

[0749] Step 1: The clone has a similar conversation to the user.

[0750] Step 2: Specifically, when a user asks, "What's the weather like today?" the clone learns to answer, "It's sunny."

[0751] "Example 3"

[0752] Step 1: The clone talks to a third party on behalf of the user.

[0753] Step 2: Specifically, the user can leave conversations with people they don't like to the clone.

[0754] Step 3: Because the clone has similar speech patterns to the user, a third party may mistakenly believe that the clone is the user.

[0755] Example 1

[0756] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0757] Conventional messenger applications lacked the ability to learn a user's speech patterns and generate a clone of the user based on them. This created a need for a system that could automatically respond when the user was absent. It was also difficult to update the user's speech patterns in real time and achieve more natural conversations.

[0758] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for receiving user utterances, means for saving the received utterances in a database, means for analyzing utterance patterns based on the saved utterance data, means for generating a clone of the user based on the analysis results, and means for updating the utterance patterns of the clone through chat between the generated clone and the user. This makes it possible to learn the user's utterance patterns, generate clones, and update utterance patterns in real time.

[0759] "User" means an individual or entity that uses the System to make statements through a messenger application.

[0760] A "statement" is a communication such as a text message or a voice message that a user sends through a messenger application.

[0761] "Means for receiving" refers to the function or process by which a user's comments are sent to a server and the server receives the comments.

[0762] A "database" is an information system that stores user utterance data and manages it so that it can be accessed and analyzed later.

[0763] "Means for saving" refers to the functions and processes for recording and storing received user comments in a database.

[0764] A "statement pattern" is a collection of data obtained by analyzing the characteristics, tendencies, frequency, context, etc. of a user's statements.

[0765] The "analysis means" refers to a function or process for extracting and analyzing a user's speech patterns based on the saved speech data.

[0766] A "clone" is a virtual agent that is generated based on the user's speech patterns and can make speeches and respond in the same way as the user.

[0767] "Means for generating" refers to the functions and processes for creating a clone of a user based on the analysis results.

[0768] "Chat" refers to the interaction or communication between a user and a clone via a messenger application.

[0769] The "means for updating" refers to a function or process by which the clone learns new speech patterns through chat with the user and improves the clone's speech patterns in real time.

[0770] This invention is a system in which a user makes statements through a messenger application. The system learns the user's speech patterns and generates a clone of the user based on the learning results. The generated clone updates its speech patterns through chats with the user.

[0771] Hardware and software used

[0772] Hardware

[0773] Server: A server equipped with a high-performance CPU and GPU

[0774] Device: The smartphone or PC used by the user

[0775] software

[0776] Generative AI models: For example, OpenAI's GPT-4

[0777] Messenger applications: for example, Slack or Microsoft Teams

[0778] Data processing and calculation

[0779] server

[0780] The server receives the user's comments and stores them in a database. Based on the stored comment data, it uses a generative AI model to analyze the user's speech patterns. Based on the analysis results, it generates a clone of the user. The generated clone updates its speech patterns through chat with the user.

[0781] Terminal

[0782] The device transmits the user's comments to the server in real time. When a user makes a comment through a messenger application, a data packet containing the comment and the user ID is generated and sent to the server.

[0783] User

[0784] A user uses a messenger application to type and send a text message, for example, the user says "Good morning."

[0785] Specific examples

[0786] 1. A user says "Good morning" in a messenger application.

[0787] 2. The device sends this message to the server.

[0788] 3. The server receives the message and stores the message "Good morning" in the database.

[0789] 4. The server uses the generative AI model to analyze the utterance pattern of "Good morning."

[0790] 5. The server generates a clone based on the analysis results.

[0791] 6. The clone replies to the user with "Good morning" and updates its speech patterns through chat with the user.

[0792] Prompt Sentence Examples

[0793] "Based on what the user says in the messenger application, learn the user's speech patterns and generate a clone. For example, if the user says 'Good morning,' the clone should also say 'Good morning.'"

[0794] In this way, a system is constructed that learns user speech patterns, generates clones, and updates speech patterns in real time.

[0795] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0796] Step 1:

[0797] A user speaks in a messenger application. The user uses the messenger application to type and send a text message. For example, the user speaks "Good morning." This utterance becomes input data.

[0798] Step 2:

[0799] The device sends the user's comments to the server. The device sends the user's comments to the server in real time. At this time, a data packet containing the comment content and the user ID is generated. The input data is the user's comment and user ID, and the output data is the data packet sent to the server.

[0800] Step 3:

[0801] The server receives the message and stores it in the database. The server receives the data packet sent from the terminal. The received message content and user ID are stored in the database. For example, the following record is added to the database:

[0802] | User ID | Post content | Timestamp |

[0803] |--------|-----------|--------------------------|

[0804] | 12345 | Good Morning | 2023-10-01 08:00:00 |

[0805] The input data are the received data packets and the output data are the records stored in the database.

[0806] Step 4:

[0807] The server analyzes speech patterns using a generative AI model. Based on the saved speech data, the server analyzes the user's speech patterns using a generative AI model (e.g., OpenAI GPT-4). The analysis results include extracting the characteristics, frequency, and context of the user's speech. The input data is the speech data saved in the database, and the output data is the analysis results.

[0808] Step 5:

[0809] The server generates a clone based on the analysis results. The server generates a clone of the user based on the analysis results. This clone is designed to mimic the user's speech patterns. For example, if the user frequently says "Good morning," the clone is set to similarly say "Good morning." The input data is the analysis results, and the output data is the generated clone.

[0810] Step 6:

[0811] The clone updates its speech patterns through chat with the user. The clone updates its speech patterns in real time through chat with the user. For example, if the user says "Good morning," the clone will also reply "Good morning." Through this interaction, the clone learns the user's new speech patterns and begins to generate more accurate responses. The input data is the content of the chat with the user, and the output data is the clone's updated speech patterns.

[0812] (Application example 1)

[0813] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0814] While existing messenger applications can learn a user's speech patterns and generate user clones, they lack the functionality to recommend content based on the user's preferences, making it difficult to efficiently provide the information and entertainment that users desire.

[0815] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0816] In this invention, the server includes means for learning a user's utterance patterns, means for generating a clone of the user based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, and means for recommending appropriate content based on the user's utterances, thereby enabling content to be recommended based on the user's preferences.

[0817] "User" means an individual or organization that uses the system.

[0818] "Speech patterns" refer to the characteristics and tendencies of statements made by users through messenger applications.

[0819] "Means of learning" refers to algorithms and programs that analyze users' speech patterns and extract their characteristics.

[0820] A "clone" is a virtual alter ego of a user that is generated based on the user's speech patterns.

[0821] "Means of generation" refers to algorithms or programs for creating clones based on learned speech patterns.

[0822] "Means for updating" refers to algorithms or programs for continuously improving the speech patterns of the generated clones through chats with users.

[0823] "Content" refers to information and entertainment such as movies, music, and news.

[0824] "Recommendation methods" refer to algorithms or programs that select and present appropriate content based on user comments.

[0825] A system for implementing this invention learns a user's speech patterns, generates a clone of the user based on the learned speech patterns, updates the clone's speech patterns through chat between the generated clone and the user, and further recommends appropriate content based on the user's speech.

[0826] System configuration

[0827] The system consists of the following main components:

[0828] 1. User device: The user uses the messenger application on a device such as a smartphone or tablet.

[0829] 2. Server: Learns the user's speech patterns, generates clones, updates the clones' speech patterns, and performs processing to recommend content.

[0830] 3. Generative AI model: Used to learn user speech patterns, generate clones, and recommend content.

[0831] Program processing

[0832] The server receives utterance data sent from the user's device and learns the utterance patterns using a generative AI model. Based on the learned utterance patterns, it generates a clone of the user. The generated clone continues to update its utterance patterns through chat with the user. Furthermore, it recommends appropriate content based on the user's utterances.

[0833] Hardware and software used

[0834] Hardware: smartphones, tablets, servers

[0835] Software: Python, OpenAI API

[0836] Data processing and calculation

[0837] Data processing: Analyze user speech data and extract speech patterns.

[0838] Data Computing: Generative AI models are used to generate clones and recommend content based on speech patterns.

[0839] Specific examples

[0840] For example, if a user says, "What movies do you recommend these days?", the server receives this utterance and uses a generative AI model to learn the user's utterance patterns. Next, it generates a clone based on the learned utterance patterns and updates the clone's utterance patterns through chat with the user. Finally, it recommends appropriate movies based on the user's utterances.

[0841] Prompt Sentence Examples

[0842] This is an assistant that learns the user's speech patterns and recommends appropriate content. Please recommend appropriate content based on the user's speech below.

[0843] User: What recent movies do you recommend?

[0844] assistant:

[0845] In this way, content can be recommended based on the user's preferences.

[0846] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0847] Step 1:

[0848] The user's terminal inputs the user's message through a messenger application.

[0849] Input: User utterance (e.g., "What are some recent movie recommendations?")

[0850] Output: Speech data

[0851] Specific operation: A user inputs a message into a messenger application, and the message data is sent to a server.

[0852] Step 2:

[0853] The server analyzes the received speech data and learns the user's speech patterns.

[0854] Input: Speech data

[0855] Output: Learned speech patterns

[0856] Specific operation: The server analyzes the speech data and extracts speech patterns using a generative AI model.

[0857] Step 3:

[0858] The server generates a clone of the user based on the learned speech patterns.

[0859] Input: Learned speech patterns

[0860] Output: Clone the user

[0861] How it works: The server uses the generative AI model to generate clones based on learned speech patterns.

[0862] Step 4:

[0863] The server updates the clone's speech patterns through chat between the clone and the user.

[0864] Input: User's new utterance, clone's utterance pattern

[0865] Output: Updated clone's utterance pattern

[0866] Specific operation: Every time the user makes a new statement, the server analyzes the statement and updates the clone's statement pattern.

[0867] Step 5:

[0868] The server recommends appropriate content based on the user's comments.

[0869] Input: User utterances, updated clone utterance patterns

[0870] Output: Recommended content

[0871] Specific operation: The server analyzes the user's comments, uses a generative AI model to select appropriate content, and recommends it to the user.

[0872] Step 6:

[0873] The user terminal displays the recommended content.

[0874] Input: Recommended content

[0875] Output: The displayed content

[0876] Specific operation: The user terminal displays the recommended content received from the server on the messenger application.

[0877] Example 2

[0878] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0879] Conventional systems have had difficulty accurately learning a user's speech patterns and generating a clone that can converse in the same way as the user. Furthermore, when the generated clone converses with a third party on behalf of the user, it sometimes fails to make appropriate remarks. This limits the usefulness of the clone as a substitute for the user.

[0880] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user utterance data, means for preprocessing the collected utterance data, means for training a generative AI model using the preprocessed data, means for generating a user clone using the trained generative AI model, and means for updating the clone's utterance pattern through chat between the generated clone and the user. This makes it possible to accurately learn the user's utterance pattern and generate a clone that can converse in a similar manner to the user. Furthermore, the generated clone can make appropriate utterances even when conversing with a third party on behalf of the user.

[0881] "User utterance data" is text data of conversations and utterances made by the user in the past.

[0882] The "collection means" refers to a method or device for acquiring and storing user utterance data.

[0883] A "preprocessing means" is a method or device for removing unnecessary information from collected speech data and arranging the text into a consistent format.

[0884] A "generative AI model" is an artificial intelligence model that learns a user's speech patterns and generates a clone that can converse in a similar way to the user.

[0885] "Means for training" refers to a method or device for training a generative AI model to learn user speech patterns using preprocessed data.

[0886] A "clone" is a virtual entity that uses a generative AI model to mimic a user's speech patterns and can have a conversation similar to that of the user.

[0887] "Means for generating" refers to a method or device for creating a clone of a user using a trained generative AI model.

[0888] The "updating means" refers to a method or device for continuously improving the speech patterns of the clone through chats between the generated clone and the user.

[0889] The present invention relates to a system that learns a user's speech patterns and generates a clone that can converse in a similar manner to the user. A specific embodiment of this system will be described below.

[0890] The server uses the API of the chat application used by the user to collect user utterance data. For example, it acquires a log of a user asking, "What's the weather like today?" This utterance data is saved in text format.

[0891] The server then preprocesses the collected speech data by removing unnecessary information (such as advertisements or system messages) and normalizing the text (such as converting uppercase to lowercase and removing special characters). This preprocessing ensures that the data is in a consistent format.

[0892] The preprocessed data is input into a generative AI model (e.g., OpenAI's GPT-3). The server uses this data to train the generative AI model. During this process, the model learns the user's speech patterns and acquires the ability to generate appropriate answers to the user's questions.

[0893] After the learning is complete, the server uses the trained generative AI model to generate a clone of the user. This clone can then have a conversation similar to the user. For example, if the user asks, "What's the weather like today?", the clone will respond, "It's sunny."

[0894] The terminal provides an interface through which the user can converse with the clone. The user types a question, and the clone responds with an appropriate answer. The interface also provides a means for the user to update the clone's speech patterns through conversation.

[0895] As a concrete example, the following prompt sentence can be input to a generative AI model:

[0896] Example prompt sentence:

[0897] User: What's the weather like today?

[0898] Crone: It's sunny.

[0899] Based on this prompt, the generative AI model generates appropriate answers to the user's questions, allowing the clone to have a conversation similar to that of the user.

[0900] This system can accurately learn a user's speech patterns and generate a clone that can converse in the same way as the user.The clone can also make appropriate remarks when conversing with a third party on behalf of the user.

[0901] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0902] Step 1: Data collection

[0903] The server collects user utterance data. Specifically, it obtains past conversation logs through the API of the chat application used by the user. These logs contain comments and questions made by the user. The input is the conversation log obtained from the chat application API, and the output is the utterance data saved in text format.

[0904] Step 2: Data Preprocessing

[0905] The server preprocesses the collected utterance data. Specifically, it performs the following processes.

[0906] Eliminate unnecessary information: Remove information that is not relevant to the conversation, such as advertisements and system messages.

[0907] Text normalization: Formatting text consistently, such as converting uppercase to lowercase and removing special characters.

[0908] The input is collected utterance data, and the output is preprocessed text data in a consistent format.

[0909] Step 3: Model training

[0910] The server inputs the preprocessed data into a generative AI model to learn the user's speech patterns. Specifically, the server supplies the preprocessed data to a generative AI model (e.g., OpenAI's GPT-3) so that the model can generate appropriate answers to the user's questions. The input is preprocessed text data, and the output is a trained generative AI model.

[0911] Step 4: Clone Generation

[0912] The server uses a trained generative AI model to generate a user clone. Specifically, the model is fed with prompts based on the user's speech patterns, and the clone generates appropriate responses. The input is the trained generative AI model and the prompts, and the output is a user clone.

[0913] Step 5: Talk to the clone

[0914] The terminal provides an interface that allows the user to converse with the clone. When the user inputs a question, the clone responds with an appropriate answer. For example, if the user inputs "What's the weather like today?", the clone responds with "It's sunny." The input is the user's question, and the output is the clone's answer.

[0915] In this way, it is possible to accurately learn a user's speech patterns and generate a clone that can converse in the same way as the user. Furthermore, the generated clone can also make appropriate statements when conversing with a third party on behalf of the user.

[0916] (Application example 2)

[0917] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0918] Conventional personal assistant systems have difficulty learning a user's speech patterns, making them unable to hold conversations similar to the user's. Furthermore, there are only a limited number of systems that can hold conversations with third parties on behalf of the user. Furthermore, they lack the ability to use generative AI models to generate responses based on user input and display them to the user. This limits the user experience and makes efficient communication difficult.

[0919] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0920] In this invention, the server includes means for learning a user's utterance patterns, means for generating a clone of the user based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for generating responses using a generative AI model based on user input, and means for displaying the generated responses to the user. This makes it possible to learn a user's utterance patterns, generate a clone that converses similarly to the user, and further generate and display responses based on the user's input using the generative AI model.

[0921] "User utterance patterns" refer to the characteristics and tendencies of a series of utterances and questions made by a user.

[0922] "Means for learning" refers to a method or device for analyzing a user's speech patterns, extracting and storing their characteristics.

[0923] A "clone" refers to a virtual entity that is generated based on the user's speech patterns and can have a conversation similar to that of the user.

[0924] "Means for generating" refers to a method or device for creating clones based on learned speech patterns.

[0925] "Chat" refers to a conversation between a user and a clone via text or voice.

[0926] "Means for updating" refers to a method or device for modifying or adding speech patterns based on new information obtained by the clone through chat with the user.

[0927] A "generative AI model" refers to an artificial intelligence model that generates appropriate responses based on user input.

[0928] "Means for generating a response" refers to a method or device for using a generative AI model to create an appropriate response to a user input.

[0929] "Means for displaying" refers to a method or device for visually or audibly presenting the generated response to the user.

[0930] A system for implementing this invention learns a user's speech patterns, generates clones, and uses a generative AI model to generate and display responses based on the user's input. A specific embodiment of this system is described below.

[0931] System Configuration

[0932] The system mainly consists of the following components:

[0933] 1. User device: A device where the user inputs data, such as a smartphone or computer.

[0934] 2. Server: A central processing unit that learns the user's speech patterns, generates clones, and generates responses using generative AI models.

[0935] 3. Generative AI models: Artificial intelligence models that generate appropriate responses based on user input, such as OpenAI's API.

[0936] Program processing

[0937] User terminal

[0938] The user terminal is responsible for receiving input from the user and sending it to the server. When the user types "What's the weather like today?", the data is sent to the server.

[0939] server

[0940] The server performs the following process:

[0941] 1. Learning speech patterns: Analyzes the user's past speech data, extracts and memorizes its features, and learns the user's speech patterns.

[0942] 2. Clone generation: Based on the learned speech patterns, a clone is generated that can have a similar conversation to the user.

[0943] 3. Use of generative AI models: Based on user input, a generative AI model (e.g., OpenAI's API) is used to generate an appropriate response.

[0944] 4. Displaying the response: The generated response is sent to the user terminal and displayed to the user.

[0945] Specific examples

[0946] When a user asks, "What's the weather like today?", the server uses the generative AI model to generate a response such as "It's sunny," and sends it to the user's device, which then displays the response to the user.

[0947] Prompt Sentence Examples

[0948] An example of a prompt sent to the generative AI model is as follows:

[0949] User: What's the weather like today?

[0950] clone:

[0951] In this way, a system is realized that learns a user's speech patterns, generates clones, and uses a generative AI model to generate and display responses based on the user's input.

[0952] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0953] Step 1:

[0954] The user terminal receives input from the user. Specifically, the user inputs "What's the weather like today?" This input data is sent to the server in text format.

[0955] Step 2:

[0956] The server analyzes the input data received from the user. The input data is analyzed and recorded as data for learning the user's speech patterns. In this step, the server also references the user's past speech data and updates the speech patterns.

[0957] Step 3:

[0958] The server generates a clone based on the learned speech patterns. The clone is a virtual entity that can have a conversation similar to the user. The generated clone reflects the user's speech patterns.

[0959] Step 4:

[0960] The server sends the user's input data to the generative AI model. Specifically, it sends a prompt to the generative AI model (e.g., OpenAI's API). An example of a prompt is "User: What's the weather like today?\nClone: ​​"

[0961] Step 5:

[0962] The generative AI model generates an appropriate response based on the prompt sentence, and the generated response is returned to the server in text format. For example, the response generated might be "It's sunny."

[0963] Step 6:

[0964] The server receives the response returned by the generative AI model and sends it to the user device. The response data is sent to the user device in text format.

[0965] Step 7:

[0966] The user terminal displays the response received from the server to the user. Specifically, the response "It's sunny" is displayed on the user's screen.

[0967] The above processing steps realize a system that learns a user's speech patterns, generates clones, and uses a generative AI model to generate and display responses based on the user's input.

[0968] Example 3

[0969] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0970] In conventional systems, there was no way for the user to take over conversations with people they did not like, which often caused stress for the user. It was also difficult to generate a clone that accurately imitated the user's speech patterns. Furthermore, there was a lack of a way to save and later check the conversation log between the clone and a third party.

[0971] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0972] In this invention, the server includes means for collecting user utterance data, means for preprocessing the collected utterance data, means for training a generative AI model using the preprocessed data, means for generating a clone of the user based on the trained generative AI model, means for using the generated clone to act on behalf of the user in a conversation with a third party, and means for saving a conversation log between the clone and the third party. This allows the user to leave conversations with people the user does not like to the clone, thereby reducing the user's stress. Furthermore, because the clone has similar utterance patterns to the user, a third party may mistakenly believe the clone to be the user. Furthermore, by saving the conversation log, the user can later check the content of the conversation.

[0973] "User" means an individual or organization that uses the system.

[0974] "Utterance data" is text information of conversations and messages that a user has had in the past.

[0975] "Preprocessing" is the process of removing noise from collected speech data and converting it into a format that is easy to analyze.

[0976] A "generative AI model" is a model that uses artificial intelligence technology to learn and imitate user speech patterns.

[0977] A "clone" is a virtual entity that mimics a user's speech patterns based on a generative AI model.

[0978] A "third party" is any person or entity other than the user or clone with whom the conversation takes place.

[0979] A "conversation log" is a record of a conversation between a clone and a third party.

[0980] "Means of collection" refers to methods and techniques for acquiring user utterance data.

[0981] "Preprocessing means" refers to methods and techniques for converting collected speech data into a format that is easy to analyze.

[0982] The "training means" refers to a method or technique for training a generative AI model using preprocessed data. The server may include means for training the generative AI model based on the updated utterance patterns.

[0983] "Means of generation" refers to methods and technologies for creating clones based on a trained generative AI model.

[0984] "Means of acting as a proxy" refers to methods or techniques for using a clone to converse with a third party.

[0985] "Means of preservation" refers to methods or technologies for recording conversation logs between the clone and a third party so that they can be reviewed later.

[0986] MODE FOR CARRYING OUT THE INVENTION

[0987] The present invention relates to a system that learns a user's speech patterns, generates a clone, and carries out a conversation with a third party on behalf of the user. A specific embodiment of this system will be described below.

[0988] System configuration

[0989] This system consists of three main components: a server, a terminal, and a user. The server collects user speech data, preprocesses it, and trains the generative AI model. The terminal uses the generated clone to converse with a third party and stores the conversation log. The user is an individual or organization that uses the system.

[0990] Hardware and software used

[0991] Server: A server with high-performance computing power is required. For example, a cloud server equipped with a GPU is suitable.

[0992] Generative AI models: Use advanced generative AI models such as OpenAI's GPT-4.

[0993] Data collection tools: Use tools to collect data from chat apps, emails, and social media via APIs, such as the Slack API and Gmail API.

[0994] Data preprocessing tools: Preprocess data using regular expressions or natural language processing libraries (e.g., NLTK or spaCy).

[0995] System Operation

[0996] The server first collects user comment data. For example, it obtains data from Slack or Gmail, which the user uses on a daily basis, via an API. The obtained data is saved as a text file.

[0997] The server then preprocesses the collected speech data, using regular expressions to remove unnecessary HTML tags and special characters, split the text into words, and standardize the case.

[0998] The server uses the preprocessed data to train a generative AI model. The preprocessed data is divided into training data (80%) and test data (20%), and the training data is trained using the GPT-4 model. Once training is complete, the model's performance is evaluated using the test data, and accuracy is confirmed to be above 90%.

[0999] Based on the trained generative AI model, the server generates a clone of the user, which is then able to make similar statements to the user by applying the user's speech patterns.

[1000] The device uses the generated clone to carry out conversations with third parties. For example, if a user entrusts a conversation with a boss they don't get along with to the clone, and the boss asks, "How's the progress this week?" the clone will respond in real time with, "It's going well. There are no particular problems." After the conversation ends, the device saves the conversation log so the user can check it later.

[1001] Examples of specific examples and prompts

[1002] Examples:

[1003] The user can leave conversations with their boss, whom they don't like, to the clone, which will then naturally converse with the boss based on the user's past utterance data.

[1004] The clone generates answers to questions from its boss that a user would normally give. For example, if the boss asks, "How is things going this week?", the clone might respond, "Everything is going well. No particular issues."

[1005] Example prompt sentence:

[1006] "Generate a clone that mimics conversations with a boss based on the user's past speech data."

[1007] "Create a program that allows the user to delegate conversations with people they don't like to a clone."

[1008] In this way, the user can leave conversations with people the user does not like to the clone. Because the clone has the same speech patterns as the user, a third party may mistakenly recognize the clone as the user. Furthermore, by saving the conversation log, the user can check the content of the conversation later. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[1009] Step 1: Collect user utterance data

[1010] The server collects user comment data. Specifically, it obtains data from chat apps, emails, and social media that users use on a daily basis. For example, it uses the Slack API or Gmail API to obtain messages from the past six months.

[1011] Input: User's chat app, email, and social networking account information

[1012] Output: Text file of acquired speech data

[1013] Step 2: Data Preprocessing

[1014] The server preprocesses the collected utterance data. Specifically, it performs the following processes.

[1015] Noise removal: Remove unnecessary HTML tags and special characters using regular expressions.

[1016] Tokenization: Dividing text data into words and phrases.

[1017] Normalization: unifying case and expanding abbreviations.

[1018] Input: Text file of acquired speech data

[1019] Output: Preprocessed text data

[1020] Step 3: Model training

[1021] The server trains the generative AI model using the preprocessed data. Specifically, it performs the following processes:

[1022] Dataset creation: Split the preprocessed data into training data (80%) and test data (20%).

[1023] Model training: A generative AI model (e.g., GPT-4) is trained on training data.

[1024] Model evaluation: Evaluate the performance of the trained model using test data.

[1025] Input: Preprocessed text data

[1026] Output: A trained generative AI model

[1027] Step 4: Clone Generation

[1028] The server generates clones using a trained generative AI model. Specifically, it performs the following processes:

[1029] Clone initialization: Initialize the clone based on the trained model.

[1030] Applying speech patterns: Applying the user's speech patterns to the clone so that it can speak in the same way as the user.

[1031] Input: A trained generative AI model

[1032] Output: A clone with the user's utterance patterns

[1033] Step 5: Clone conversation

[1034] The terminal uses the clone to act as a proxy for the conversation with the third party. Specifically, it performs the following processes.

[1035] Start conversation: The device launches the clone and starts a conversation with the third party.

[1036] Real-time response: The clone responds to questions and comments from third parties in real time based on the user's speech patterns.

[1037] Saving conversation logs: The device saves conversation logs between the clone and the third party, allowing the user to review them later.

[1038] Input: Clone with user's speech patterns, questions and statements from third parties

[1039] Output: Conversation log between clone and third party

[1040] In this way, users can leave conversations with people they don't like to the clone. Because the clone has the same speech patterns as the user, a third party may mistakenly believe that the clone is the user. Furthermore, by saving the conversation log, users can check the content of the conversation later.

[1041] (Application example 3)

[1042] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1043] Conventional clone generation systems only imitate the user's speech patterns, and it is difficult to achieve the accuracy required to mislead third parties in real conversations. Furthermore, when the clone takes over a conversation with someone or in a situation that the user is not comfortable with, the clone's responses often become unnatural. Furthermore, the application of clones to security checks and authentication processes has not been sufficiently considered.

[1044] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for learning a user's utterance patterns, means for generating a user clone based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for generating the clone's responses using a generative AI model, and means for generating prompt sentences and inputting the clone's responses to the generative AI model. This makes it possible to generate a clone that imitates a user's utterance patterns with high accuracy and has such natural conversations that it can mislead third parties. Furthermore, the application of clones to perform security checks and authentication processes is realized.

[1045] "User speech patterns" refer to the characteristics of the words and speaking style that a user uses on a daily basis.

[1046] "Means for learning" refers to a method or device for analyzing a user's speech patterns and storing their characteristics as data.

[1047] A "clone" refers to an artificial intelligence that imitates a user's speech patterns and converses on their behalf.

[1048] "Means for generating" refers to a method or device for creating a clone based on the learned speech patterns of a user.

[1049] The "means for updating" refers to a method or device for continuously improving the speech patterns of the clone through chat between the generated clone and the user.

[1050] "Generative AI model" refers to a model for generating text using artificial intelligence.

[1051] "Means for generating responses" refers to a method or device for using a generative AI model to generate the clone's utterances.

[1052] A "prompt sentence" refers to an instruction sentence to be input into a generative AI model.

[1053] "Means for input" refers to a method or device for passing a prompt sentence to a generative AI model and generating a response.

[1054] A system for implementing this invention learns a user's speech patterns, generates a clone based on those patterns, and updates the clone's speech patterns through chat between the generated clone and the user. It also includes means for generating responses for the clone using a generative AI model, generating prompt sentences, and inputting the clone's responses to the generative AI model.

[1055] To learn the user's speech patterns, the server collects the user's past conversation data and analyzes it using natural language processing technology, specifically using software such as Python and TensorFlow to model the user's speech patterns.

[1056] The server then generates a clone based on the learned speech patterns. The clone can then mimic the user's speech patterns and engage in conversations similar to the user's. The clone is generated using OpenAI's GPT-3 model.

[1057] The generated clones update their speech patterns through chats with the user, allowing them to always reflect the latest user speech patterns. The updated speech patterns are then fed back into the model to improve the accuracy of the clones.

[1058] In addition, the server uses a generative AI model to generate responses for the clone. Specifically, it generates prompt sentences and inputs them into the GPT-3 model to generate responses for the clone. Examples of prompt sentences include the following:

[1059] Example prompt sentence:

[1060] Generate the following conversation based on the user's speech patterns:

[1061] User profile: The user is a man in his 30s who works in the technical field. He speaks politely and logically.

[1062] User Input: Hello, how can I help you today?

[1063] Clone response:

[1064] Using this prompt, the GPT-3 model generates a response that mimics the user's speech pattern, for example, "Hello, do you need technical support today?"

[1065] This system makes it possible to generate clones that can accurately mimic a user's speech patterns and converse so naturally that it can be easily mistaken for a third party. It also enables the application of clones to perform security checks and authentication processes.

[1066] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1067] Step 1:

[1068] The server collects the user's past conversation data. Specifically, it acquires text data such as chat applications and emails used by the user. The input is the user's past conversation data, and the output is text data for analysis.

[1069] Step 2:

[1070] The server analyzes the collected conversation data using natural language processing technology. Specifically, it uses Python and TensorFlow to model the user's speech patterns. The input is the collected text data, and the output is a model that represents the user's speech patterns.

[1071] Step 3:

[1072] The server generates clones based on the learned speech patterns. Specifically, it uses OpenAI's GPT-3 model to create clones that mimic the user's speech patterns. The input is a model that represents the user's speech patterns, and the output is the generated clone.

[1073] Step 4:

[1074] The server updates the clone's speech patterns through chats between the generated clone and the user. Specifically, it collects chat data between the user and the clone and feeds it back into the model. The input is the chat data between the user and the clone, and the output is a model that represents the updated speech patterns.

[1075] Step 5:

[1076] The server generates the clone's response using a generative AI model. Specifically, it generates a prompt sentence and inputs it into the GPT-3 model to generate the clone's response. The input is the prompt sentence, and the output is the clone's response.

[1077] Step 6:

[1078] The server sends the clone's response to the user or a third party. Specifically, it sends the response via a chat application or email. The input is the clone's response, and the output is the message sent to the user or a third party.

[1079] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1080] "Example 1"

[1081] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system learns a user's speech patterns and generates a clone of the user based on the learned speech patterns. The clone's speech patterns are updated through chats between the generated clone and the user. Furthermore, this system includes an emotion engine that recognizes the user's emotions.

[1082] "Example 2"

[1083] The emotion engine recognizes emotions from the user's words and actions. For example, if a user says, "Today was fun," the emotion engine recognizes from this statement that the user is feeling happy. This recognition result is used to update the clone's speech patterns.

[1084] "Example 3"

[1085] Specifically, if the emotion engine recognizes joy, the clone is updated to make similar statements in situations that make the user feel joy. For example, the clone will say, "Today was fun." This allows the clone to make statements that reflect the user's emotions.

[1086] "Example 4"

[1087] The emotion engine is also used by the clone to adjust its conversation with third parties. For example, if the emotion engine recognizes that the user is feeling angry, the clone will use that information to appropriately adjust its conversation with third parties. Specifically, in situations where the user feels angry, the clone will make statements that express the user's anger.

[1088] The processing flow of each embodiment will be described below.

[1089] "Example 1"

[1090] Step 1: The system initiates a chat with the user through a messaging app.

[1091] Step 2: The system learns the user's speech patterns through chat.

[1092] Step 3: Based on the learned speech patterns, the system generates a clone of the user.

[1093] Step 4: Through chat between the generated clone and the user, the system updates the clone's speech patterns.

[1094] Step 5: The system activates the emotion engine to recognize the user's emotions.

[1095] "Example 2"

[1096] Step 1: The emotion engine recognizes emotions from the user's statements and actions.

[1097] Step 2: The recognized emotions are used to update the clone's speech patterns.

[1098] "Example 3"

[1099] Step 1: When the emotion engine recognizes a particular emotion, the clone's speech pattern is updated to reflect that emotion.

[1100] Step 2: Based on the updated speech patterns, the clone makes speech that expresses similar emotions to the user.

[1101] "Example 4"

[1102] Step 1: The emotion engine recognizes the user's emotion.

[1103] Step 2: Based on the perceived emotion, the clone adjusts the conversation with the third party.

[1104] Step 3: Specifically, the clone makes statements that reflect the emotions the user might be feeling.

[1105] Example 1

[1106] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1107] Conventional chat systems can learn a user's speech patterns and generate a user clone, but they lack the ability to recognize the user's emotions and generate responses accordingly. This makes it difficult to communicate in a way that is sensitive to the user's emotions, limiting the user experience.

[1108] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for collecting user utterance data, means for preprocessing the collected utterance data, means for learning utterance patterns using the preprocessed utterance data, means for generating a user clone based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for recognizing emotions from the user's utterances, and means for generating responses according to the recognized emotions. This makes it possible to learn the user's utterance patterns and generate responses according to the emotions.

[1109] The "means of collecting user comment data" refers to a function that saves comments made by users through the messenger app as a log and sends them to a server.

[1110] The "means for preprocessing collected utterance data" is a function that converts collected utterance data into a format that is easy to analyze by processing the data such as tokenizing, removing stop words, and stemming.

[1111] The "means for learning utterance patterns using preprocessed utterance data" is a function that uses a machine learning algorithm based on preprocessed data to analyze user utterance patterns and train a model.

[1112] The "means for generating a clone of a user based on learned speech patterns" is a function for creating a virtual agent that mimics the speech patterns of a user based on a learned model.

[1113] The "means for updating the clone's speech patterns through chat between the generated clone and the user" is a function that continuously learns and updates the clone's speech patterns using newly collected speech data through chat between the clone and the user.

[1114] The "means for recognizing emotions from user utterances" is a function that analyzes the content of a user's utterances and identifies the user's emotions using an emotion engine.

[1115] The "means for generating a response according to the recognized emotion" is a function for generating an appropriate response based on the user's emotion identified by the emotion engine.

[1116] This invention is a system that learns a user's speech patterns and generates a clone of the user based on the learning results. Furthermore, the generated clone updates its speech patterns through chats with the user, recognizes the user's emotions using an emotion engine, and generates responses accordingly.

[1117] The server uses a messenger app to collect user speech data. All speech made by users within the app is saved as a log and sent to the server. The collected speech data is preprocessed using natural language processing libraries such as Python's NLTK and spaCy. Preprocessing includes tokenizing the text data, removing stop words, and stemming.

[1118] The preprocessed speech data is used to learn speech patterns using machine learning frameworks such as TensorFlow and PyTorch. Specifically, a recurrent neural network (RNN) or transformer model is constructed, the user's speech patterns are analyzed, and the model is trained. A clone of the user is generated based on the learned model. The clone is a virtual agent that mimics the user's speech patterns.

[1119] The device then starts a chat between the user and the clone. When the user speaks through a messaging app, the clone responds accordingly. For example, if the user says "Good morning," the clone responds with "Good morning!"

[1120] Newly collected speech data from chats with the clone is sent to the server and used to update the clone's speech patterns, allowing the clone to reflect the user's latest speech patterns.

[1121] Furthermore, the device uses an emotion engine to recognize emotions from the user's speech. The emotion engine uses Microsoft Azure's Text Analytics API and IBM Watson's Tone Analyzer, among others. Once the emotion engine recognizes the user's emotion, it generates an appropriate response based on that emotion. For example, if the user says, "I'm tired today," the emotion engine will recognize the user's sense of fatigue and respond, "Thank you for your hard work. Please take a good rest today."

[1122] Example prompt sentence:

[1123] User: Good morning

[1124] Clone: ​​Good morning!

[1125] User: I'm tired today

[1126] Clone: ​​Thank you for your hard work. Please get some rest today.

[1127] In this way, the server, terminal, and user work together to learn the user's speech patterns, generate clones, and respond according to their emotions.

[1128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1129] Step 1: Collect user utterance data

[1130] The server collects user speech data through the messenger app. When a user speaks within the app, the speech is saved as a log and sent to the server. The input is the user's speech text, and the output is the speech log saved on the server. Specifically, when a user says "Good morning," the text data is sent to the server.

[1131] Step 2: Preprocessing the utterance data

[1132] The server preprocesses the collected utterance data. The input is the collected utterance data, and the output is preprocessed text data. Specifically, it uses Python's NLTK and spaCy to tokenize the text data, remove stop words, and perform stemming. For example, the utterance "Good morning" is tokenized and unnecessary characters are removed.

[1133] Step 3: Learning speech patterns

[1134] The server uses preprocessed utterance data to learn utterance patterns. The input is preprocessed text data, and the output is a trained model. Specifically, a recurrent neural network (RNN) or transformer model is built using TensorFlow or PyTorch, and the user's utterance patterns are analyzed to train the model. For example, the utterance pattern "Good morning" is learned by the model.

[1135] Step 4: Creating a clone

[1136] The server generates a user clone based on the trained model. The input is the trained model, and the output is the user clone. The clone is a virtual agent that imitates the user's speech patterns. For example, a clone that says "Good morning" is generated.

[1137] Step 5: Chat with the clone

[1138] The device starts a chat between the generated clone and the user. The input is the user's speech, and the output is the clone's response. When the user speaks through a messenger app, the clone responds accordingly. For example, if the user says "Good morning," the clone responds "Good morning!"

[1139] Step 6: Update the clone's speech patterns

[1140] The server updates the clone's speech patterns using newly collected speech data through chats with the clone. The input is the newly collected speech data, and the output is the updated model. This allows the clone to reflect the user's latest speech patterns. For example, when a user says, "I'm tired today," that data is sent to the server, and the clone's speech patterns are updated.

[1141] Step 7: Emotion recognition and response generation by the emotion engine

[1142] The device uses an emotion engine to recognize emotions from what the user says. The input is what the user says, and the output is the recognized emotion and an appropriate response. Specifically, it uses Microsoft Azure's Text Analytics API and IBM Watson's Tone Analyzer. Once the emotion engine recognizes the user's emotion, it generates an appropriate response based on that emotion. For example, if the user says, "I'm tired today," the emotion engine will recognize the user's sense of fatigue and respond, "Thank you for your hard work. Please take it easy and rest today."

[1143] (Application example 1)

[1144] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1145] Conventional user interfaces and content recommendation systems have had difficulty fully understanding a user's speech patterns and emotions and providing individually optimized content. Furthermore, there were no systems that could converse with a third party on the user's behalf, which placed a heavy burden on the user.

[1146] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for learning a user's speech patterns, means for generating a user clone based on the learned speech patterns, means for updating the clone's speech patterns through chat between the generated clone and the user, means for recognizing the user's emotions, and means for recommending appropriate content based on the recognized emotions. This makes it possible to understand the user's speech patterns and emotions and provide individually optimized content. It is also possible to have a conversation with a third party on behalf of the user.

[1147] "User speech patterns" refers to the tendencies and characteristics of speech made by users through messenger apps and chat systems.

[1148] "Means of learning" refers to algorithms and systems that analyze users' speech patterns and accumulate their characteristics as data.

[1149] A "clone" is a virtual entity that is generated based on a user's speech patterns and imitates the same speech and behavior as the user.

[1150] "Means for updating the clone's speech patterns through chat" refers to a function that enables the clone to learn new speech patterns through dialogue with the user and to modify and update its own speech patterns.

[1151] "Means for recognizing emotions" refers to algorithms or systems that analyze emotions from a user's words and actions and identify their emotional state.

[1152] "Means for recommending appropriate content" refers to a function for selecting and providing the most suitable content (movies, music, articles, etc.) to a user based on the user's recognized emotions.

[1153] As an embodiment of the present invention, a system is constructed that learns user speech patterns, generates clones, recognizes emotions, and recommends appropriate content. A specific embodiment of this system is shown below.

[1154] System Configuration

[1155] The system consists of the following main components:

[1156] 1. User device: A device such as a smartphone or computer that users use to input their comments.

[1157] 2. Server: A central processing unit for learning user speech patterns, generating clones, recognizing emotions, and recommending content.

[1158] 3. Emotion Recognition Engine: Software that uses emotion analysis tools such as TextBlob to recognize emotions from user speech.

[1159] 4. Generative AI model: An algorithm that uses OpenAI's API to generate clone responses based on the user's speech patterns.

[1160] Program processing

[1161] The server performs the following process.

[1162] 1. A means of learning user speech patterns: Analyzes speech sent from the user's device and stores the speech patterns in a database. This allows us to understand the user's speech tendencies.

[1163] 2. Method for generating clones: Based on the learned speech patterns, a generative AI model is used to generate a clone of the user, which can make similar speech to the user.

[1164] 3. A method for updating the clone's speech patterns: Through chats between the clone and the user, new speech patterns are learned and the clone's speech patterns are updated.

[1165] 4. Means of Emotion Recognition: An emotion recognition engine is used to analyze emotions from the user's speech and identify their emotional state.

[1166] 5. Means of recommending appropriate content: Based on the recognized emotions, the most suitable content (movies, music, articles, etc.) for the user is selected and provided to the user's device.

[1167] Hardware and software used

[1168] Hardware: smartphones, PCs, servers

[1169] Software: Python, OpenAI API, TextBlob

[1170] Specific examples

[1171] If the user says "I'm so happy today!", the following happens:

[1172] 1. A message is sent from the user's device to the server.

[1173] 2. The server learns speech patterns and stores them in a database.

[1174] 3. Using a generative AI model, the clone responds, "That's great!"

[1175] 4. The emotion recognition engine analyzes the user's speech and recognizes positive emotions.

[1176] 5. The server recommends content based on positive emotions, such as "Here is a happy movie for you!"

[1177] Prompt Sentence Examples

[1178] User says: I'm so happy today!

[1179] AI Clone responds:

[1180] In this way, a system is realized that understands the user's speech patterns and emotions and provides individually optimized content.

[1181] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1182] Step 1:

[1183] A message is sent from the user terminal to the server.

[1184] Input: User utterance (e.g., "I'm so happy today!")

[1185] Processing: The user terminal sends the utterance as text data to the server.

[1186] Output: Speech data arrives at the server.

[1187] Step 2:

[1188] The server learns speech patterns and stores them in a database.

[1189] Input: User's speech data

[1190] Processing: The server analyzes the speech data and extracts speech patterns. The extracted patterns are stored in a database.

[1191] Output: Updated utterance pattern database

[1192] Step 3:

[1193] Using a generative AI model, the clone generates a response.

[1194] Input: User utterance data, utterance pattern database

[1195] Processing: The server uses the OpenAI API to generate a clone's response based on the user's utterance. The server inputs the prompt sentence into the generative AI model and obtains the response.

[1196] Output: The clone's response (e.g., "That's great!")

[1197] Step 4:

[1198] The clone's response is sent to the user terminal.

[1199] Input: Clone response

[1200] Processing: The server sends the generated clone response to the user terminal.

[1201] Output: The clone's response displayed on the user's terminal

[1202] Step 5:

[1203] The emotion recognition engine analyzes the user's speech and recognizes their emotions.

[1204] Input: User's speech data

[1205] Processing: The server uses TextBlob to parse the sentiment from the user's speech and identify positive, negative, and neutral sentiment states.

[1206] Output: Recognized emotion data (e.g., positive)

[1207] Step 6:

[1208] The server recommends appropriate content based on the recognized emotions.

[1209] Input: Recognized emotion data

[1210] Processing: The server selects the most suitable content for the user based on the emotion data. For example, if the emotion is positive, it will select content such as "Here is a happy movie for you!"

[1211] Output: Recommended content

[1212] Step 7:

[1213] The recommended content is transmitted to the user terminal.

[1214] Input: Recommended content

[1215] Processing: The server sends the recommended content to the user terminal.

[1216] Output: Recommended content displayed on the user's device

[1217] In this way, a system is realized that understands the user's speech patterns and emotions and provides individually optimized content.

[1218] Example 2

[1219] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1220] Conventional systems were able to learn the user's speech patterns and generate clones, but they were unable to recognize the user's emotions and dynamically update the clone's speech patterns based on those emotions. As a result, conversations with the user were one-sided, making it difficult to have natural dialogue. Furthermore, the inability to respond to the user's emotions led to a poor user experience.

[1221] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for learning a user's utterance patterns, means for generating a clone of the user based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for recognizing emotions from the user's utterances and actions, and means for updating the clone's utterance patterns based on the recognized emotions. This enables natural conversation according to the user's emotions.

[1222] The "means for learning user speech patterns" is a function that analyzes speech made by a user, extracts and records their characteristics and tendencies.

[1223] The "means for generating a user clone" is a function that creates a virtual entity that makes statements similar to those of the user based on the learned speech patterns of the user.

[1224] The "means for updating the clone's speech patterns" is a function that improves the content of the clone's speech by reflecting new information and speech patterns that the generated clone has acquired through chat with the user.

[1225] "Means for recognizing emotions from user's statements and actions" is a function that analyzes the statements and actions made by the user and identifies the emotions behind them.

[1226] "Means for updating the clone's speech patterns based on recognized emotions" is a function that adjusts the content and tone of the clone's speech based on the results of emotion recognition, thereby realizing more natural and emotionally appropriate dialogue.

[1227] MODE FOR CARRYING OUT THE INVENTION

[1228] This invention is a system that learns a user's speech patterns, generates a clone, and updates the clone's speech patterns through dialogue with the user. It can also recognize emotions from the user's speech and behavior, and dynamically update the clone's speech patterns based on those emotions.

[1229] Hardware and software used

[1230] Hardware: Servers, terminals (PCs, smartphones)

[1231] Software: Generative AI models (e.g., GPT-4), emotion recognition engines (e.g., IBM Watson Tone Analyzer)

[1232] Specific operation of the system

[1233] 1. Accepting user input

[1234] The device receives the user's comments and questions in text format.

[1235] Example: A user types "What's the weather like today?" into a chat app on their smartphone.

[1236] The terminal transmits this input to the server in real time.

[1237] 2. Answer Generation by Cloning

[1238] The server receives user input from the device and sends it to the generative AI model.

[1239] Example: The server sends the user input "What's the weather like today?" as a prompt to a generative AI model (e.g., GPT-4).

[1240] A generative AI model generates appropriate answers based on user input.

[1241] Example: The generative AI model generates the answer "It's sunny."

[1242] The server sends the generated response to the terminal.

[1243] The terminal displays the answer to the user.

[1244] Example: The device displays "It's sunny" in a chat app.

[1245] 3. Emotional Recognition

[1246] The server sends the user's utterances to the emotion recognition engine.

[1247] Example: If a user says, "I had fun today," the server sends this statement to an emotion recognition engine (e.g., IBM Watson Tone Analyzer).

[1248] The emotion recognition engine analyzes emotions from user speech.

[1249] Example: An emotion recognition engine recognizes "joy" from the statement "Today was fun."

[1250] The server receives the results of the emotion recognition engine.

[1251] Example: The server receives the recognition result "joy" from the emotion recognition engine.

[1252] 4. Updating the clone's speech patterns

[1253] The server updates the clone's speech patterns based on the results of the emotion recognition engine.

[1254] Example: Based on the emotion recognition result of "joy," the server updates the clone's speech pattern in a positive direction.

[1255] Set the clone to say something positive in the next conversation, such as "Today was fun!"

[1256] Prompt Sentence Examples

[1257] "When a user asks, 'What's the weather like today?' how do you respond?"

[1258] "When a user says, 'I had fun today,' what emotion do you recognize?"

[1259] In this way, the system can have a natural conversation with the user and respond according to the user's emotions.

[1260] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1261] Step 1:

[1262] Accepting user input

[1263] The device receives the user's comments and questions in text format.

[1264] Input: A user types "What's the weather like today?" into a chat app on their smartphone.

[1265] Specific operation: The device sends this input to the server in real time.

[1266] Output: User's text input sent to the server.

[1267] Step 2:

[1268] Answer generation by cloning

[1269] The server receives user input from the device and sends it to the generative AI model.

[1270] Input: The server sends the user input "What's the weather like today?" as a prompt to the generative AI model.

[1271] What happens: A generative AI model (e.g., GPT-4) generates an appropriate answer based on user input.

[1272] Output: The answer generated by the generative AI model: "It's sunny."

[1273] Step 3:

[1274] Submitting and viewing answers

[1275] The server sends the generated response to the terminal.

[1276] Input: The answer from the generative AI model: "It's sunny."

[1277] Specific operation: The server receives the answer from the generative AI model and sends it to the device.

[1278] The terminal displays the answer to the user.

[1279] Output: The device displays "It's sunny" in the chat app.

[1280] Step 4:

[1281] Emotion recognition

[1282] The server sends the user's utterances to the emotion recognition engine.

[1283] Input: Text data when a user says "Today was fun."

[1284] Specific operation: The server sends this utterance to an emotion recognition engine (e.g., IBM Watson Tone Analyzer).

[1285] The emotion recognition engine analyzes emotions from user speech.

[1286] Output: The emotion "joy" recognized by the emotion recognition engine.

[1287] Step 5:

[1288] Receiving emotion recognition results

[1289] The server receives the results of the emotion recognition engine.

[1290] Input: The recognition result "joy" from the emotion recognition engine.

[1291] What it does: The server receives the results from the emotion recognition engine and uses them to update the clone's speech patterns.

[1292] Output: Data for updating the clone's utterance patterns.

[1293] Step 6:

[1294] Updated clone speech patterns

[1295] The server updates the clone's speech patterns based on the results of the emotion recognition engine.

[1296] Input: Emotion recognition result "Joy".

[1297] Specific operation: Based on the emotion recognition result of "joy," the server updates the clone's speech pattern in a positive direction.

[1298] Output: Updated clone utterance patterns.

[1299] In this way, the system can have a natural conversation with the user and respond according to the user's emotions.

[1300] (Application example 2)

[1301] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1302] Conventional systems that learn user speech patterns and generate clones were unable to recognize the user's emotions and respond appropriately based on them. This made it difficult to provide appropriate service through dialogue with customers, especially in physical stores. Furthermore, there was a lack of a way to recognize customer emotions in real time and provide feedback to store staff, making it difficult to improve customer satisfaction.

[1303] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1304] In this invention, the server includes means for learning a user's speech patterns, means for generating a clone of the user based on the learned speech patterns, means for updating the clone's speech patterns through chat between the generated clone and the user, means for recognizing the user's emotions, means for updating the clone's speech patterns based on the recognized emotions, and means for recognizing customer emotions in real time and supporting appropriate responses, thereby making it possible to recognize customer emotions in real time and provide appropriate feedback.

[1305] "Means for learning user speech patterns" refers to technology for analyzing the characteristics and trends of user speech and accumulating the data.

[1306] "Means for generating a user clone based on learned speech patterns" is a technology that creates a virtual entity (clone) that makes similar statements to the user based on the accumulated speech patterns of the user.

[1307] "Means for updating the clone's speech patterns through chat between the generated clone and the user" refers to a technology that continuously improves and updates the clone's speech patterns through dialogue between the clone and the user.

[1308] "Means for recognizing user emotions" refers to technology for analyzing and identifying emotions from a user's statements and actions.

[1309] "Means for updating the clone's speech patterns based on the recognized emotions" refers to a technology that appropriately adjusts and updates the clone's speech patterns based on the results of analyzing the user's emotions.

[1310] "Means to recognize customer emotions in real time and support appropriate responses" refers to technology that analyzes customer comments and facial expressions in real time, and instructs and supports store staff on appropriate responses based on those emotions.

[1311] The system for implementing this invention has the functions of learning a user's speech patterns, generating a clone, and recognizing the user's emotions to update the clone's speech patterns. A specific embodiment of this system will be described below.

[1312] First, the server collects and analyzes user utterances to learn their speech patterns. This is done by using natural language processing technology to extract the characteristics and trends of the utterances. Examples of software used include NLTK and spaCy, which are natural language processing libraries for Python.

[1313] Next, based on the learned speech patterns, the server generates a clone of the user. This clone is a virtual entity that speaks in a similar way to the user and is created using a generative AI model, specifically, OpenAI's GPT-3.

[1314] The generated clone updates its speech patterns through chats with the user. The server collects data on the clone's interactions with the user and continuously improves the clone's speech patterns. This process is performed using a machine learning algorithm.

[1315] Furthermore, the server analyzes the user's speech and behavior to recognize the user's emotions. For emotion recognition, it uses Transformers emotion analysis models. For example, it uses the Hugging Face Transformers library to identify emotions from the user's speech.

[1316] Based on the recognized emotion, the server updates the clone's speech patterns, allowing the clone to make appropriate speech according to the user's emotion.

[1317] In a physical store, when a salesperson wearing smart glasses talks to a customer, the glasses recognize the customer's emotions in real time and provide feedback to the salesperson. The smart glasses are equipped with a camera that detects the customer's face using OpenCV and analyzes their emotions using a Transformers emotion recognition model.

[1318] For example, if a customer says, "I really like this product!", the server recognizes that the customer is happy and notifies the store clerk, "The customer is happy. Please suggest additional products."

[1319] An example of a prompt to input to a generative AI model is as follows:

[1320] What customers say: "I absolutely love this product!"

[1321] Emotion recognition result: "POSITIVE"

[1322] Feedback: "Customers are happy. Please suggest additional products."

[1323] In this way, the system can recognize customer emotions in real time and provide appropriate feedback.

[1324] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1325] Step 1:

[1326] The server collects user utterances. It receives the text data uttered by the user as input and uses natural language processing technology to analyze the characteristics and trends of the utterances. Specifically, it uses Python natural language processing libraries NLTK and spaCy to tokenize the utterances, tag parts of speech, and perform grammatical analysis. The analysis results are stored in a database as the user's utterance patterns.

[1327] Step 2:

[1328] The server generates a clone of the user based on the learned speech patterns. It uses the speech patterns collected and analyzed in step 1 as input. It uses a generative AI model (e.g., OpenAI's GPT-3) to create a virtual entity (clone) that speaks similarly to the user. The generated clone mimics the user's speech patterns.

[1329] Step 3:

[1330] The server updates the clone's speech patterns through chats between the generated clone and the user. It collects interaction data between the clone and the user as input. It analyzes the interaction data and applies a machine learning algorithm to continuously improve and update the clone's speech patterns. The updated speech patterns are reflected in the clone's speech model.

[1331] Step 4:

[1332] The server recognizes the user's emotions. It uses the user's speech and behavior data as input. For emotion recognition, it uses a Transformers emotion analysis model (for example, the Hugging Face Transformers library). It identifies emotions from the user's speech and saves the results as emotion data.

[1333] Step 5:

[1334] The server updates the clone's speech patterns based on the recognized emotions. It uses the emotion data recognized in step 4 as input. Based on the emotion data, it adjusts and updates the clone's speech patterns appropriately. This allows the clone to make appropriate speech according to the user's emotions.

[1335] Step 6:

[1336] The device (smart glasses) recognizes the customer's emotions in real time and supports appropriate responses. It uses the customer's facial image captured by the smart glasses' camera as input. It detects the customer's face using OpenCV and analyzes their emotions using a Transformers emotion recognition model. The analysis results are then sent to the store clerk as feedback.

[1337] Step 7:

[1338] The terminal (smart glasses) displays the feedback to the store clerk. It uses the emotion recognition results analyzed in step 6 as input. For example, if the customer says, "I really like this product!", the server recognizes that the customer is happy and notifies the store clerk, "The customer is happy. Please suggest additional products." The notification is displayed on the smart glasses display.

[1339] Example 3

[1340] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1341] Conventional systems were able to learn a user's speech patterns and generate a clone, but they had difficulty generating speech that reflected the user's emotions or appropriately adjusting conversations with third parties. Furthermore, they lacked a mechanism for updating the model based on user feedback, making it difficult to improve the accuracy of the clone's speech. This resulted in the clone being unable to converse naturally on behalf of the user, resulting in a poor user experience.

[1342] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1343] In this invention, the server includes means for learning a user's utterance patterns, means for generating a user clone based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for collecting user utterance data, means for analyzing emotions based on the collected utterance data, means for generating utterances for the clone based on the analysis results, means for using the utterances of the generated clone in conversations with third parties, and means for updating the model based on feedback. This makes it possible to generate natural utterances that reflect the user's emotions and appropriately adjust conversations with third parties. Furthermore, by updating the model based on user feedback, the accuracy of the clone's utterances can be improved.

[1344] "Means for learning user utterance patterns" refers to technology or devices that analyze past utterances made by users and extract their characteristics and trends.

[1345] "Means for generating a user clone" refers to technology or devices that create a virtual entity that makes statements similar to those of the user, based on the learned speech patterns of the user.

[1346] "Means for updating the clone's speech patterns" refers to technology or devices that allow the generated clone to learn new speech patterns through chats with users and improve and update the content of its speech.

[1347] "Means for collecting user utterance data" refers to technology or devices for recording voice or text utterances made by users and saving them as data.

[1348] "Means for analyzing emotions" refers to technology or devices for identifying and analyzing emotions from collected user speech data.

[1349] "Means for generating clone utterances" refers to technology or devices that generate text or audio for the clone to make appropriate utterances based on the results of emotion analysis.

[1350] "Means used for conversation with a third party" refers to the technology or device used to apply the utterances of the generated clone to a conversation with a third party and actually carry out the conversation.

[1351] "Means for updating the model based on feedback" refers to technologies and devices that retrain the generative AI model based on user evaluations and feedback, improving the accuracy of the clone's speech.

[1352] The present invention relates to a system that learns a user's speech patterns, generates a clone that reflects the user's emotions, and has a conversation with a third party. A specific embodiment of this system will be described below.

[1353] First, users generate utterance data through everyday conversations and text messages. The device collects this utterance data in real time and sends it to a server. Specifically, devices such as smartphones and PCs record and record the user's voice and text, and upload them to the server.

[1354] Next, the server inputs the received utterance data into the emotion engine. The emotion engine uses natural language processing technology to analyze the emotion from the user's utterance. Specifically, it uses a general emotion analysis tool. For example, if a user says, "Today was fun," the emotion engine recognizes "joy."

[1355] The server then inputs a prompt sentence into the generative AI model based on the analysis results of the emotion engine. The generative AI model generates the clone's speech that reflects the user's speech patterns and emotions. Specifically, a general generative AI model is used. For example, the prompt sentence "User's speech: Today was fun. Emotion: Joy" is input, and the clone's speech is generated as "Today was really fun."

[1356] The generated clone's speech is used in conversations with third parties. The device outputs the clone's speech as voice using speech synthesis technology. Specifically, a common speech synthesis tool is used. For example, the clone might say, "I had a really good time today," and the device would play that speech.

[1357] Furthermore, users provide feedback on the clone's comments and the results of the conversation. The server updates the generative AI model based on this feedback, improving the accuracy of the clone's comments. Specifically, if a user evaluates a comment as "appropriate," the data is added to the model's training data.

[1358] As a concrete example, consider the case where a user says "Today was fun" in a conversation with a friend. The device collects this utterance data and sends it to the server. The server uses an emotion engine to analyze "joy" and inputs the prompt text "User utterance: Today was fun. Emotion: Joy" into the generative AI model. The generative AI model generates the clone's utterance, "Today was really fun," and the device plays back that audio.

[1359] Example prompt sentence:

[1360] User says: "Today was fun"

[1361] Emotion engine analysis result: "Joy"

[1362] Clone says: "I had a really good time today."

[1363] In this way, the server analyzes the user's emotions and speech patterns and adjusts the clone to make appropriate speech. This makes it possible to generate natural speech that reflects the user's emotions and appropriately adjust conversations with third parties. In addition, by updating the model based on user feedback, the accuracy of the clone's speech can be improved. The flow of the identification process in Example 3 will be described with reference to FIG. 21.

[1364] Step 1:

[1365] Users generate utterance data through everyday conversations and text messages. Devices collect this utterance data in real time and send it to a server. Specifically, devices such as smartphones and PCs record and record users' voices and text, and upload them to a server. The input is the user's utterance data, and the output is the utterance data sent to the server.

[1366] Step 2:

[1367] The server inputs the received utterance data into the emotion engine. The emotion engine uses natural language processing technology to analyze emotions from the user's utterances. Specifically, it uses a general emotion analysis tool. For example, if a user utters, "I had fun today," the emotion engine recognizes "joy." The input is the utterance data, and the output is analyzed emotion data.

[1368] Step 3:

[1369] The server inputs a prompt sentence into the generative AI model based on the analysis results of the emotion engine. The generative AI model generates the clone's utterances that reflect the user's utterance patterns and emotions. Specifically, a general generative AI model is used. For example, the prompt sentence is input as "User's utterance: Today was fun. Emotion: Joy," and the clone's utterance is generated as "Today was really fun." The input is the prompt sentence, and the output is the clone's utterance.

[1370] Step 4:

[1371] The generated clone's utterances are used in conversations with third parties. The device outputs the clone's utterances as voice using speech synthesis technology. Specifically, a common speech synthesis tool is used. For example, the clone may say, "I had a really good time today," and the device plays back the voice. The input is the clone's utterance, and the output is the utterance played back as voice.

[1372] Step 5:

[1373] The user provides feedback on the clone's utterances and the results of the conversation. The server updates the generative AI model based on this feedback, improving the accuracy of the clone's utterances. Specifically, if the user evaluates a utterance as "appropriate," the data is added to the model's training data. The input is the user's feedback, and the output is an updated generative AI model.

[1374] (Application example 3)

[1375] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1376] Conventional systems that learn a user's speech patterns have difficulty responding appropriately when the user faces unpleasant or dangerous situations. Furthermore, they lack the ability to recognize the user's emotions and adjust speech based on those emotions, which can lead to inappropriate responses when conversing with a third party on the user's behalf. This can potentially compromise the user's safety and comfort.

[1377] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1378] In this invention, the server includes means for learning a user's speech patterns, means for generating a clone of the user based on the learned speech patterns, means for updating the clone's speech patterns through chat between the generated clone and the user, means for recognizing the user's emotions, means for adjusting the clone's speech based on the recognized emotions, and means for the clone to take an appropriate response when the user faces an unpleasant or dangerous situation. This enables an appropriate response that reflects the user's emotions, thereby improving the user's safety and comfort.

[1379] "User speech patterns" refer to the words, phrases, and speaking style characteristics that a user uses on a daily basis.

[1380] A "clone" refers to an artificial intelligence that learns a user's speech patterns and converses on their behalf.

[1381] "Chat" refers to text-based interactive communication.

[1382] "Means for recognizing emotions" refers to technology that analyzes emotions from a user's statements and actions and identifies specific emotions.

[1383] "Means to tailor speech" refers to technology that alters the content and tone of a clone's speech based on perceived emotions.

[1384] An "unpleasant situation" refers to a situation in which the user feels psychologically uncomfortable.

[1385] A "dangerous situation" refers to a situation in which the user feels physically or psychologically unsafe.

[1386] "Appropriate response" refers to actions and statements necessary to maintain the safety and comfort of users.

[1387] The system for implementing this invention has the functions of learning a user's speech patterns, generating a clone of the user, and updating the clone's speech patterns through chat between the user and the generated clone. It also has the functions of recognizing the user's emotions, adjusting the clone's speech based on the recognized emotions, and allowing the clone to respond appropriately when the user faces an unpleasant or dangerous situation.

[1388] System Configuration

[1389] The system consists of the following major components:

[1390] 1. User terminal: A device such as a smartphone or tablet used by the user. It provides an interface for inputting user comments and chatting with the clone.

[1391] 2. Server: Learns the user's speech patterns, generates clones, recognizes emotions, and processes the clones' speech to adjust them.

[1392] 3. Emotion recognition engine: Software that analyzes emotions from a user's words and actions and identifies specific emotions.

[1393] 4. Clone generation engine: Software that generates clones based on the user's speech patterns and updates the clones' speech patterns.

[1394] Program processing

[1395] The server receives the speech data sent from the user terminal and analyzes the user's emotions using an emotion recognition engine. Based on the analysis results, the clone generation engine adjusts the clone's speech and takes appropriate action on behalf of the user.

[1396] Specifically, if a user says, "I'm having fun today," the emotion recognition engine will recognize "joy," and the clone generation engine will respond with, "The user is feeling happy." In this way, it is possible to respond appropriately to the user's emotions.

[1397] Hardware and software used

[1398] Hardware: smartphones, tablets, servers

[1399] Software: Emotion recognition engine, clone generation engine

[1400] Specific examples

[1401] For example, if the user inputs "Today is fun," the system operates as follows:

[1402] 1. The user terminal sends the speech data to the server.

[1403] 2. The server uses an emotion recognition engine to recognize "joy."

[1404] 3. The clone generation engine responds, "The user finds it fun."

[1405] Prompt Sentence Examples

[1406] When a user says "I'm having fun today," generate a program in which the emotion engine recognizes "joy" and the clone responds "The user is feeling happy."

[1407] In this way, it is possible to respond appropriately to the user's feelings, thereby improving the safety and comfort of the user.

[1408] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1409] Step 1:

[1410] The user terminal inputs speech data.

[1411] A user uses a smartphone or tablet to input a statement. For example, they might type, "Today is fun." The input data is sent to the server in text format.

[1412] Step 2:

[1413] The server receives the speech data.

[1414] The server receives the utterance data sent from the user terminal and temporarily stores the received data for use in the next processing step.

[1415] Step 3:

[1416] The server analyzes the speech data using an emotion recognition engine.

[1417] The server inputs the received utterance data into an emotion recognition engine. The emotion recognition engine analyzes the text data and identifies the user's emotion. For example, it recognizes "joy" from an utterance such as "Today is fun." The analysis results are output as emotion data.

[1418] Step 4:

[1419] The server generates clone utterances using a clone generation engine.

[1420] The server inputs the emotion data output from the emotion recognition engine into the clone generation engine. The clone generation engine generates appropriate utterances based on the user's utterance patterns and emotion data. For example, if "joy" is recognized, the engine generates the utterance "The user is feeling happy." The generated utterance is output as text data.

[1421] Step 5:

[1422] The server transmits the generated utterance data to the user terminal.

[1423] The server transmits the utterance data output from the clone generation engine to the user terminal, where it is displayed.

[1424] Step 6:

[1425] The user terminal displays the generated utterance data.

[1426] The user terminal receives the utterance data sent from the server and displays it on the screen. For example, a utterance such as "The user is having fun" is displayed. This allows the user to check the clone's response.

[1427] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing.

[1428] The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1429] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1430] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1431] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1432] [Third embodiment]

[1433] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1434] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1435] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1436] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1437] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20 .

[1438] The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1440] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1441] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1442] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1443] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1444] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1445] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1446] "Example 1"

[1447] One embodiment of the present invention is a system in which a user makes statements through a messenger app. This system learns the user's speech patterns and generates a clone of the user based on the learning results. The generated clone updates its speech patterns through chats with the user. Specifically, if the user says "Good morning," the clone also learns to say "Good morning."

[1448] "Example 2"

[1449] Furthermore, the clone can have a conversation similar to that of the user. For example, if the user asks, "What's the weather like today?", the clone will learn to respond, "It's sunny."

[1450] "Example 3"

[1451] The clone can also converse with third parties on behalf of the user. Specifically, the user can leave conversations with people the clone does not like to talk to. Because the clone has the same speech patterns as the user, third parties may mistake the clone for the user.

[1452] The processing flow of each embodiment will be described below.

[1453] "Example 1"

[1454] Step 1: The user makes a comment through the messenger app.

[1455] Step 2: The system learns the user's speech patterns.

[1456] Step 3: Generate a clone of the user based on the learning results.

[1457] Step 4: The generated clone updates its speech patterns through chat with the user. Specifically, if the user says "Good morning," the clone also learns to say "Good morning."

[1458] "Example 2"

[1459] Step 1: The clone has a similar conversation to the user.

[1460] Step 2: Specifically, when a user asks, "What's the weather like today?" the clone learns to answer, "It's sunny."

[1461] "Example 3"

[1462] Step 1: The clone talks to a third party on behalf of the user.

[1463] Step 2: Specifically, the user can leave conversations with people they don't like to the clone.

[1464] Step 3: Because the clone has similar speech patterns to the user, a third party may mistakenly believe that the clone is the user.

[1465] Example 1

[1466] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1467] Conventional messenger applications lacked the ability to learn a user's speech patterns and generate a clone of the user based on them. This created a need for a system that could automatically respond when the user was absent. It was also difficult to update the user's speech patterns in real time and achieve more natural conversations.

[1468] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for receiving user utterances, means for saving the received utterances in a database, means for analyzing utterance patterns based on the saved utterance data, means for generating a clone of the user based on the analysis results, and means for updating the utterance patterns of the clone through chat between the generated clone and the user. This makes it possible to learn the user's utterance patterns, generate clones, and update utterance patterns in real time.

[1469] "User" means an individual or entity that uses the System to make statements through a messenger application.

[1470] A "statement" is a communication such as a text message or a voice message that a user sends through a messenger application.

[1471] "Means for receiving" refers to the function or process by which a user's comments are sent to a server and the server receives the comments.

[1472] A "database" is an information system that stores user utterance data and manages it so that it can be accessed and analyzed later.

[1473] "Means for saving" refers to the functions and processes for recording and storing received user comments in a database.

[1474] A "statement pattern" is a collection of data obtained by analyzing the characteristics, tendencies, frequency, context, etc. of a user's statements.

[1475] The "analysis means" refers to a function or process for extracting and analyzing a user's speech patterns based on the saved speech data.

[1476] A "clone" is a virtual agent that is generated based on the user's speech patterns and can make speeches and respond in the same way as the user.

[1477] "Means for generating" refers to the functions and processes for creating a clone of a user based on the analysis results.

[1478] "Chat" refers to the interaction or communication between a user and a clone via a messenger application.

[1479] The "means for updating" refers to a function or process by which the clone learns new speech patterns through chat with the user and improves the clone's speech patterns in real time.

[1480] This invention is a system in which a user makes statements through a messenger application. The system learns the user's speech patterns and generates a clone of the user based on the learning results. The generated clone updates its speech patterns through chats with the user.

[1481] Hardware and software used

[1482] Hardware

[1483] Server: A server equipped with a high-performance CPU and GPU

[1484] Device: The smartphone or PC used by the user

[1485] software

[1486] Generative AI models: For example, OpenAI's GPT-4

[1487] Messenger applications: for example, Slack or Microsoft Teams

[1488] Data processing and calculation

[1489] server

[1490] The server receives the user's comments and stores them in a database. Based on the stored comment data, it uses a generative AI model to analyze the user's speech patterns. Based on the analysis results, it generates a clone of the user. The generated clone updates its speech patterns through chat with the user.

[1491] Terminal

[1492] The device transmits the user's comments to the server in real time. When a user makes a comment through a messenger application, a data packet containing the comment and the user ID is generated and sent to the server.

[1493] User

[1494] A user uses a messenger application to type and send a text message, for example, the user says "Good morning."

[1495] Specific examples

[1496] 1. A user says "Good morning" in a messenger application.

[1497] 2. The device sends this message to the server.

[1498] 3. The server receives the message and stores the message "Good morning" in the database.

[1499] 4. The server uses the generative AI model to analyze the utterance pattern of "Good morning."

[1500] 5. The server generates a clone based on the analysis results.

[1501] 6. The clone replies to the user with "Good morning" and updates its speech patterns through chat with the user.

[1502] Prompt Sentence Examples

[1503] "Based on what the user says in the messenger application, learn the user's speech patterns and generate a clone. For example, if the user says 'Good morning,' the clone should also say 'Good morning.'"

[1504] In this way, a system is constructed that learns user speech patterns, generates clones, and updates speech patterns in real time.

[1505] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1506] Step 1:

[1507] A user speaks in a messenger application. The user uses the messenger application to type and send a text message. For example, the user speaks "Good morning." This utterance becomes input data.

[1508] Step 2:

[1509] The device sends the user's comments to the server. The device sends the user's comments to the server in real time. At this time, a data packet containing the comment content and the user ID is generated. The input data is the user's comment and user ID, and the output data is the data packet sent to the server.

[1510] Step 3:

[1511] The server receives the message and stores it in the database. The server receives the data packet sent from the terminal. The received message content and user ID are stored in the database. For example, the following record is added to the database:

[1512] | User ID | Post content | Timestamp |

[1513] |--------|-----------|--------------------------|

[1514] | 12345 | Good Morning | 2023-10-01 08:00:00 |

[1515] The input data are the received data packets and the output data are the records stored in the database.

[1516] Step 4:

[1517] The server analyzes speech patterns using a generative AI model. Based on the saved speech data, the server analyzes the user's speech patterns using a generative AI model (e.g., OpenAI GPT-4). The analysis results include extracting the characteristics, frequency, and context of the user's speech. The input data is the speech data saved in the database, and the output data is the analysis results.

[1518] Step 5:

[1519] The server generates a clone based on the analysis results. The server generates a clone of the user based on the analysis results. This clone is designed to mimic the user's speech patterns. For example, if the user frequently says "Good morning," the clone is set to similarly say "Good morning." The input data is the analysis results, and the output data is the generated clone.

[1520] Step 6:

[1521] The clone updates its speech patterns through chat with the user. The clone updates its speech patterns in real time through chat with the user. For example, if the user says "Good morning," the clone will also reply "Good morning." Through this interaction, the clone learns the user's new speech patterns and begins to generate more accurate responses. The input data is the content of the chat with the user, and the output data is the clone's updated speech patterns.

[1522] (Application example 1)

[1523] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1524] While existing messenger applications can learn a user's speech patterns and generate user clones, they lack the functionality to recommend content based on the user's preferences, making it difficult to efficiently provide the information and entertainment that users desire.

[1525] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1526] In this invention, the server includes means for learning a user's utterance patterns, means for generating a clone of the user based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, and means for recommending appropriate content based on the user's utterances, thereby enabling content to be recommended based on the user's preferences.

[1527] "User" means an individual or organization that uses the system.

[1528] "Speech patterns" refer to the characteristics and tendencies of statements made by users through messenger applications.

[1529] "Means of learning" refers to algorithms and programs that analyze users' speech patterns and extract their characteristics.

[1530] A "clone" is a virtual alter ego of a user that is generated based on the user's speech patterns.

[1531] "Means of generation" refers to algorithms or programs for creating clones based on learned speech patterns.

[1532] "Means for updating" refers to algorithms or programs for continuously improving the speech patterns of the generated clones through chats with users.

[1533] "Content" refers to information and entertainment such as movies, music, and news.

[1534] "Recommendation methods" refer to algorithms or programs that select and present appropriate content based on user comments.

[1535] A system for implementing this invention learns a user's speech patterns, generates a clone of the user based on the learned speech patterns, updates the clone's speech patterns through chat between the generated clone and the user, and further recommends appropriate content based on the user's speech.

[1536] System configuration

[1537] The system consists of the following main components:

[1538] 1. User device: The user uses the messenger application on a device such as a smartphone or tablet.

[1539] 2. Server: Learns the user's speech patterns, generates clones, updates the clones' speech patterns, and performs processing to recommend content.

[1540] 3. Generative AI model: Used to learn user speech patterns, generate clones, and recommend content.

[1541] Program processing

[1542] The server receives utterance data sent from the user's device and learns the utterance patterns using a generative AI model. Based on the learned utterance patterns, it generates a clone of the user. The generated clone continues to update its utterance patterns through chat with the user. Furthermore, it recommends appropriate content based on the user's utterances.

[1543] Hardware and software used

[1544] Hardware: smartphones, tablets, servers

[1545] Software: Python, OpenAI API

[1546] Data processing and calculation

[1547] Data processing: Analyze user speech data and extract speech patterns.

[1548] Data Computing: Generative AI models are used to generate clones and recommend content based on speech patterns.

[1549] Specific examples

[1550] For example, if a user says, "What movies do you recommend these days?", the server receives this utterance and uses a generative AI model to learn the user's utterance patterns. Next, it generates a clone based on the learned utterance patterns and updates the clone's utterance patterns through chat with the user. Finally, it recommends appropriate movies based on the user's utterances.

[1551] Prompt Sentence Examples

[1552] This is an assistant that learns the user's speech patterns and recommends appropriate content. Please recommend appropriate content based on the user's speech below.

[1553] User: What recent movies do you recommend?

[1554] assistant:

[1555] In this way, content can be recommended based on the user's preferences.

[1556] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1557] Step 1:

[1558] The user's terminal inputs the user's message through a messenger application.

[1559] Input: User utterance (e.g., "What are some recent movie recommendations?")

[1560] Output: Speech data

[1561] Specific operation: A user inputs a message into a messenger application, and the message data is sent to a server.

[1562] Step 2:

[1563] The server analyzes the received speech data and learns the user's speech patterns.

[1564] Input: Speech data

[1565] Output: Learned speech patterns

[1566] Specific operation: The server analyzes the speech data and extracts speech patterns using a generative AI model.

[1567] Step 3:

[1568] The server generates a clone of the user based on the learned speech patterns.

[1569] Input: Learned speech patterns

[1570] Output: Clone the user

[1571] How it works: The server uses the generative AI model to generate clones based on learned speech patterns.

[1572] Step 4:

[1573] The server updates the clone's speech patterns through chat between the clone and the user.

[1574] Input: User's new utterance, clone's utterance pattern

[1575] Output: Updated clone's utterance pattern

[1576] Specific operation: Every time the user makes a new statement, the server analyzes the statement and updates the clone's statement pattern.

[1577] Step 5:

[1578] The server recommends appropriate content based on the user's comments.

[1579] Input: User utterances, updated clone utterance patterns

[1580] Output: Recommended content

[1581] Specific operation: The server analyzes the user's comments, uses a generative AI model to select appropriate content, and recommends it to the user.

[1582] Step 6:

[1583] The user terminal displays the recommended content.

[1584] Input: Recommended content

[1585] Output: The displayed content

[1586] Specific operation: The user terminal displays the recommended content received from the server on the messenger application.

[1587] Example 2

[1588] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1589] Conventional systems have had difficulty accurately learning a user's speech patterns and generating a clone that can converse in the same way as the user. Furthermore, when the generated clone converses with a third party on behalf of the user, it sometimes fails to make appropriate remarks. This limits the usefulness of the clone as a substitute for the user.

[1590] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user utterance data, means for preprocessing the collected utterance data, means for training a generative AI model using the preprocessed data, means for generating a user clone using the trained generative AI model, and means for updating the clone's utterance pattern through chat between the generated clone and the user. This makes it possible to accurately learn the user's utterance pattern and generate a clone that can converse in a similar manner to the user. Furthermore, the generated clone can make appropriate utterances even when conversing with a third party on behalf of the user.

[1591] "User utterance data" is text data of conversations and utterances made by the user in the past.

[1592] The "collection means" refers to a method or device for acquiring and storing user utterance data.

[1593] A "preprocessing means" is a method or device for removing unnecessary information from collected speech data and arranging the text into a consistent format.

[1594] A "generative AI model" is an artificial intelligence model that learns a user's speech patterns and generates a clone that can converse in a similar way to the user.

[1595] "Means for training" refers to a method or device for training a generative AI model to learn user speech patterns using preprocessed data.

[1596] A "clone" is a virtual entity that uses a generative AI model to mimic a user's speech patterns and can have a conversation similar to that of the user.

[1597] "Means for generating" refers to a method or device for creating a clone of a user using a trained generative AI model.

[1598] The "updating means" refers to a method or device for continuously improving the speech patterns of the clone through chats between the generated clone and the user.

[1599] The present invention relates to a system that learns a user's speech patterns and generates a clone that can converse in a similar manner to the user. A specific embodiment of this system will be described below.

[1600] The server uses the API of the chat application used by the user to collect user utterance data. For example, it acquires a log of a user asking, "What's the weather like today?" This utterance data is saved in text format.

[1601] The server then preprocesses the collected speech data by removing unnecessary information (such as advertisements or system messages) and normalizing the text (such as converting uppercase to lowercase and removing special characters). This preprocessing ensures that the data is in a consistent format.

[1602] The preprocessed data is input into a generative AI model (e.g., OpenAI's GPT-3). The server uses this data to train the generative AI model. During this process, the model learns the user's speech patterns and acquires the ability to generate appropriate answers to the user's questions.

[1603] After the learning is complete, the server uses the trained generative AI model to generate a clone of the user. This clone can then have a conversation similar to the user. For example, if the user asks, "What's the weather like today?", the clone will respond, "It's sunny."

[1604] The terminal provides an interface through which the user can converse with the clone. The user types a question, and the clone responds with an appropriate answer. The interface also provides a means for the user to update the clone's speech patterns through conversation.

[1605] As a concrete example, the following prompt sentence can be input to a generative AI model:

[1606] Example prompt sentence:

[1607] User: What's the weather like today?

[1608] Crone: It's sunny.

[1609] Based on this prompt, the generative AI model generates appropriate answers to the user's questions, allowing the clone to have a conversation similar to that of the user.

[1610] This system can accurately learn a user's speech patterns and generate a clone that can converse in the same way as the user.The clone can also make appropriate remarks when conversing with a third party on behalf of the user.

[1611] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1612] Step 1: Data collection

[1613] The server collects user utterance data. Specifically, it obtains past conversation logs through the API of the chat application used by the user. These logs contain comments and questions made by the user. The input is the conversation log obtained from the chat application API, and the output is the utterance data saved in text format.

[1614] Step 2: Data Preprocessing

[1615] The server preprocesses the collected utterance data. Specifically, it performs the following processes.

[1616] Eliminate unnecessary information: Remove information that is not relevant to the conversation, such as advertisements and system messages.

[1617] Text normalization: Formatting text consistently, such as converting uppercase to lowercase and removing special characters.

[1618] The input is collected utterance data, and the output is preprocessed text data in a consistent format.

[1619] Step 3: Model training

[1620] The server inputs the preprocessed data into a generative AI model to learn the user's speech patterns. Specifically, the server supplies the preprocessed data to a generative AI model (e.g., OpenAI's GPT-3) so that the model can generate appropriate answers to the user's questions. The input is preprocessed text data, and the output is a trained generative AI model.

[1621] Step 4: Clone Generation

[1622] The server uses a trained generative AI model to generate a user clone. Specifically, the model is fed with prompts based on the user's speech patterns, and the clone generates appropriate responses. The input is the trained generative AI model and the prompts, and the output is a user clone.

[1623] Step 5: Talk to the clone

[1624] The terminal provides an interface that allows the user to converse with the clone. When the user inputs a question, the clone responds with an appropriate answer. For example, if the user inputs "What's the weather like today?", the clone responds with "It's sunny." The input is the user's question, and the output is the clone's answer.

[1625] In this way, it is possible to accurately learn a user's speech patterns and generate a clone that can converse in the same way as the user. Furthermore, the generated clone can also make appropriate statements when conversing with a third party on behalf of the user.

[1626] (Application example 2)

[1627] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1628] Conventional personal assistant systems have difficulty learning a user's speech patterns, making them unable to hold conversations similar to the user's. Furthermore, there are only a limited number of systems that can hold conversations with third parties on behalf of the user. Furthermore, they lack the ability to use generative AI models to generate responses based on user input and display them to the user. This limits the user experience and makes efficient communication difficult.

[1629] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1630] In this invention, the server includes means for learning a user's utterance patterns, means for generating a clone of the user based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for generating responses using a generative AI model based on user input, and means for displaying the generated responses to the user. This makes it possible to learn a user's utterance patterns, generate a clone that converses similarly to the user, and further generate and display responses based on the user's input using the generative AI model.

[1631] "User utterance patterns" refer to the characteristics and tendencies of a series of utterances and questions made by a user.

[1632] "Means for learning" refers to a method or device for analyzing a user's speech patterns, extracting and storing their characteristics.

[1633] A "clone" refers to a virtual entity that is generated based on the user's speech patterns and can have a conversation similar to that of the user.

[1634] "Means for generating" refers to a method or device for creating clones based on learned speech patterns.

[1635] "Chat" refers to a conversation between a user and a clone via text or voice.

[1636] "Means for updating" refers to a method or device for modifying or adding speech patterns based on new information obtained by the clone through chat with the user.

[1637] A "generative AI model" refers to an artificial intelligence model that generates appropriate responses based on user input.

[1638] "Means for generating a response" refers to a method or device for using a generative AI model to create an appropriate response to a user input.

[1639] "Means for displaying" refers to a method or device for visually or audibly presenting the generated response to the user.

[1640] A system for implementing this invention learns a user's speech patterns, generates clones, and uses a generative AI model to generate and display responses based on the user's input. A specific embodiment of this system is described below.

[1641] System Configuration

[1642] The system mainly consists of the following components:

[1643] 1. User device: A device where the user inputs data, such as a smartphone or computer.

[1644] 2. Server: A central processing unit that learns the user's speech patterns, generates clones, and generates responses using generative AI models.

[1645] 3. Generative AI models: Artificial intelligence models that generate appropriate responses based on user input, such as OpenAI's API.

[1646] Program processing

[1647] User terminal

[1648] The user terminal is responsible for receiving input from the user and sending it to the server. When the user types "What's the weather like today?", the data is sent to the server.

[1649] server

[1650] The server performs the following process:

[1651] 1. Learning speech patterns: Analyzes the user's past speech data, extracts and memorizes its features, and learns the user's speech patterns.

[1652] 2. Clone generation: Based on the learned speech patterns, a clone is generated that can have a similar conversation to the user.

[1653] 3. Use of generative AI models: Based on user input, a generative AI model (e.g., OpenAI's API) is used to generate an appropriate response.

[1654] 4. Displaying the response: The generated response is sent to the user terminal and displayed to the user.

[1655] Specific examples

[1656] When a user asks, "What's the weather like today?", the server uses the generative AI model to generate a response such as "It's sunny," and sends it to the user's device, which then displays the response to the user.

[1657] Prompt Sentence Examples

[1658] An example of a prompt sent to the generative AI model is as follows:

[1659] User: What's the weather like today?

[1660] clone:

[1661] In this way, a system is realized that learns a user's speech patterns, generates clones, and uses a generative AI model to generate and display responses based on the user's input.

[1662] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1663] Step 1:

[1664] The user terminal receives input from the user. Specifically, the user inputs "What's the weather like today?" This input data is sent to the server in text format.

[1665] Step 2:

[1666] The server analyzes the input data received from the user. The input data is analyzed and recorded as data for learning the user's speech patterns. In this step, the server also references the user's past speech data and updates the speech patterns.

[1667] Step 3:

[1668] The server generates a clone based on the learned speech patterns. The clone is a virtual entity that can have a conversation similar to the user. The generated clone reflects the user's speech patterns.

[1669] Step 4:

[1670] The server sends the user's input data to the generative AI model. Specifically, it sends a prompt to the generative AI model (e.g., OpenAI's API). An example of a prompt is "User: What's the weather like today?\nClone: ​​"

[1671] Step 5:

[1672] The generative AI model generates an appropriate response based on the prompt sentence, and the generated response is returned to the server in text format. For example, the response generated might be "It's sunny."

[1673] Step 6:

[1674] The server receives the response returned by the generative AI model and sends it to the user device. The response data is sent to the user device in text format.

[1675] Step 7:

[1676] The user terminal displays the response received from the server to the user. Specifically, the response "It's sunny" is displayed on the user's screen.

[1677] The above processing steps realize a system that learns a user's speech patterns, generates clones, and uses a generative AI model to generate and display responses based on the user's input.

[1678] Example 3

[1679] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1680] In conventional systems, there was no way for the user to take over conversations with people they did not like, which often caused stress for the user. It was also difficult to generate a clone that accurately imitated the user's speech patterns. Furthermore, there was a lack of a way to save and later check the conversation log between the clone and a third party.

[1681] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1682] In this invention, the server includes means for collecting user utterance data, means for preprocessing the collected utterance data, means for training a generative AI model using the preprocessed data, means for generating a clone of the user based on the trained generative AI model, means for using the generated clone to act on behalf of the user in a conversation with a third party, and means for saving a conversation log between the clone and the third party. This allows the user to leave conversations with people the user does not like to the clone, thereby reducing the user's stress. Furthermore, because the clone has similar utterance patterns to the user, a third party may mistakenly believe the clone to be the user. Furthermore, by saving the conversation log, the user can later check the content of the conversation.

[1683] "User" means an individual or organization that uses the system.

[1684] "Utterance data" is text information of conversations and messages that a user has had in the past.

[1685] "Preprocessing" is the process of removing noise from collected speech data and converting it into a format that is easy to analyze.

[1686] A "generative AI model" is a model that uses artificial intelligence technology to learn and imitate user speech patterns.

[1687] A "clone" is a virtual entity that mimics a user's speech patterns based on a generative AI model.

[1688] A "third party" is any person or entity other than the user or clone with whom the conversation takes place.

[1689] A "conversation log" is a record of a conversation between a clone and a third party.

[1690] "Means of collection" refers to methods and techniques for acquiring user utterance data.

[1691] "Preprocessing means" refers to methods and techniques for converting collected speech data into a format that is easy to analyze.

[1692] A "training tool" is a method or technique for training a generative AI model using preprocessed data.

[1693] "Means of generation" refers to methods and technologies for creating clones based on a trained generative AI model.

[1694] "Means of acting as a proxy" refers to methods or techniques for using a clone to converse with a third party.

[1695] "Means of preservation" refers to methods or technologies for recording conversation logs between the clone and a third party so that they can be reviewed later.

[1696] MODE FOR CARRYING OUT THE INVENTION

[1697] The present invention relates to a system that learns a user's speech patterns, generates a clone, and carries out a conversation with a third party on behalf of the user. A specific embodiment of this system will be described below.

[1698] System configuration

[1699] This system consists of three main components: a server, a terminal, and a user. The server collects user speech data, preprocesses it, and trains the generative AI model. The terminal uses the generated clone to converse with a third party and stores the conversation log. The user is an individual or organization that uses the system.

[1700] Hardware and software used

[1701] Server: A server with high-performance computing power is required. For example, a cloud server equipped with a GPU is suitable.

[1702] Generative AI models: Use advanced generative AI models such as OpenAI's GPT-4.

[1703] Data collection tools: Use tools to collect data from chat apps, emails, and social media via APIs, such as the Slack API and Gmail API.

[1704] Data preprocessing tools: Preprocess data using regular expressions or natural language processing libraries (e.g., NLTK or spaCy).

[1705] System Operation

[1706] The server first collects user comment data. For example, it obtains data from Slack or Gmail, which the user uses on a daily basis, via an API. The obtained data is saved as a text file.

[1707] The server then preprocesses the collected speech data, using regular expressions to remove unnecessary HTML tags and special characters, split the text into words, and standardize the case.

[1708] The server uses the preprocessed data to train a generative AI model. The preprocessed data is divided into training data (80%) and test data (20%), and the training data is trained using the GPT-4 model. Once training is complete, the model's performance is evaluated using the test data, and accuracy is confirmed to be above 90%.

[1709] Based on the trained generative AI model, the server generates a clone of the user, which is then able to make similar statements to the user by applying the user's speech patterns.

[1710] The device uses the generated clone to carry out conversations with third parties. For example, if a user entrusts a conversation with a boss they don't get along with to the clone, and the boss asks, "How's the progress this week?" the clone will respond in real time with, "It's going well. There are no particular problems." After the conversation ends, the device saves the conversation log so the user can check it later.

[1711] Examples of specific examples and prompts

[1712] Examples:

[1713] The user can leave conversations with their boss, whom they don't like, to the clone, which will then naturally converse with the boss based on the user's past utterance data.

[1714] The clone generates answers to questions from its boss that a user would normally give. For example, if the boss asks, "How is things going this week?", the clone might respond, "Everything is going well. No particular issues."

[1715] Example prompt sentence:

[1716] "Generate a clone that mimics conversations with a boss based on the user's past speech data."

[1717] "Create a program that allows the user to delegate conversations with people they don't like to a clone."

[1718] In this way, the user can leave conversations with people the user does not like to the clone. Because the clone has the same speech patterns as the user, a third party may mistakenly recognize the clone as the user. Furthermore, by saving the conversation log, the user can check the content of the conversation later. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[1719] Step 1: Collect user utterance data

[1720] The server collects user comment data. Specifically, it obtains data from chat apps, emails, and social media that users use on a daily basis. For example, it uses the Slack API or Gmail API to obtain messages from the past six months.

[1721] Input: User's chat app, email, and social networking account information

[1722] Output: Text file of acquired speech data

[1723] Step 2: Data Preprocessing

[1724] The server preprocesses the collected utterance data. Specifically, it performs the following processes.

[1725] Noise removal: Remove unnecessary HTML tags and special characters using regular expressions.

[1726] Tokenization: Dividing text data into words and phrases.

[1727] Normalization: unifying case and expanding abbreviations.

[1728] Input: Text file of acquired speech data

[1729] Output: Preprocessed text data

[1730] Step 3: Model training

[1731] The server trains the generative AI model using the preprocessed data. Specifically, it performs the following processes:

[1732] Dataset creation: Split the preprocessed data into training data (80%) and test data (20%).

[1733] Model training: A generative AI model (e.g., GPT-4) is trained on training data.

[1734] Model evaluation: Evaluate the performance of the trained model using test data.

[1735] Input: Preprocessed text data

[1736] Output: A trained generative AI model

[1737] Step 4: Clone Generation

[1738] The server generates clones using a trained generative AI model. Specifically, it performs the following processes:

[1739] Clone initialization: Initialize the clone based on the trained model.

[1740] Applying speech patterns: Applying the user's speech patterns to the clone so that it can speak in the same way as the user.

[1741] Input: A trained generative AI model

[1742] Output: A clone with the user's utterance patterns

[1743] Step 5: Clone conversation

[1744] The terminal uses the clone to act as a proxy for the conversation with the third party. Specifically, it performs the following processes.

[1745] Start conversation: The device launches the clone and starts a conversation with the third party.

[1746] Real-time response: The clone responds to questions and comments from third parties in real time based on the user's speech patterns.

[1747] Saving conversation logs: The device saves conversation logs between the clone and the third party, allowing the user to review them later.

[1748] Input: Clone with user's speech patterns, questions and statements from third parties

[1749] Output: Conversation log between clone and third party

[1750] In this way, users can leave conversations with people they don't like to the clone. Because the clone has the same speech patterns as the user, a third party may mistakenly believe that the clone is the user. Furthermore, by saving the conversation log, users can check the content of the conversation later.

[1751] (Application example 3)

[1752] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1753] Conventional clone generation systems only imitate the user's speech patterns, and it is difficult to achieve the accuracy required to mislead third parties in real conversations. Furthermore, when the clone takes over a conversation with someone or in a situation that the user is not comfortable with, the clone's responses often become unnatural. Furthermore, the application of clones to security checks and authentication processes has not been sufficiently considered.

[1754] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for learning a user's utterance patterns, means for generating a user clone based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for generating the clone's responses using a generative AI model, and means for generating prompt sentences and inputting the clone's responses to the generative AI model. This makes it possible to generate a clone that imitates a user's utterance patterns with high accuracy and has such natural conversations that it can mislead third parties. Furthermore, the application of clones to perform security checks and authentication processes is realized.

[1755] "User speech patterns" refer to the characteristics of the words and speaking style that a user uses on a daily basis.

[1756] "Means for learning" refers to a method or device for analyzing a user's speech patterns and storing their characteristics as data.

[1757] A "clone" refers to an artificial intelligence that imitates a user's speech patterns and converses on their behalf.

[1758] "Means for generating" refers to a method or device for creating a clone based on the learned speech patterns of a user.

[1759] The "means for updating" refers to a method or device for continuously improving the speech patterns of the clone through chat between the generated clone and the user.

[1760] "Generative AI model" refers to a model for generating text using artificial intelligence.

[1761] "Means for generating responses" refers to a method or device for using a generative AI model to generate the clone's utterances.

[1762] A "prompt sentence" refers to an instruction sentence to be input into a generative AI model.

[1763] "Means for input" refers to a method or device for passing a prompt sentence to a generative AI model and generating a response.

[1764] A system for implementing this invention learns a user's speech patterns, generates a clone based on those patterns, and updates the clone's speech patterns through chat between the generated clone and the user. It also includes means for generating responses for the clone using a generative AI model, generating prompt sentences, and inputting the clone's responses to the generative AI model.

[1765] To learn the user's speech patterns, the server collects the user's past conversation data and analyzes it using natural language processing technology, specifically using software such as Python and TensorFlow to model the user's speech patterns.

[1766] The server then generates a clone based on the learned speech patterns. The clone can then mimic the user's speech patterns and engage in conversations similar to the user's. The clone is generated using OpenAI's GPT-3 model.

[1767] The generated clones update their speech patterns through chats with the user, allowing them to always reflect the latest user speech patterns. The updated speech patterns are then fed back into the model to improve the accuracy of the clones.

[1768] In addition, the server uses a generative AI model to generate responses for the clone. Specifically, it generates prompt sentences and inputs them into the GPT-3 model to generate responses for the clone. Examples of prompt sentences include the following:

[1769] Example prompt sentence:

[1770] Generate the following conversation based on the user's speech patterns:

[1771] User profile: The user is a man in his 30s who works in the technical field. He speaks politely and logically.

[1772] User Input: Hello, how can I help you today?

[1773] Clone response:

[1774] Using this prompt, the GPT-3 model generates a response that mimics the user's speech pattern, for example, "Hello, do you need technical support today?"

[1775] This system makes it possible to generate clones that can accurately mimic a user's speech patterns and converse so naturally that it can be easily mistaken for a third party. It also enables the application of clones to perform security checks and authentication processes.

[1776] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1777] Step 1:

[1778] The server collects the user's past conversation data. Specifically, it acquires text data such as chat applications and emails used by the user. The input is the user's past conversation data, and the output is text data for analysis.

[1779] Step 2:

[1780] The server analyzes the collected conversation data using natural language processing technology. Specifically, it uses Python and TensorFlow to model the user's speech patterns. The input is the collected text data, and the output is a model that represents the user's speech patterns.

[1781] Step 3:

[1782] The server generates clones based on the learned speech patterns. Specifically, it uses OpenAI's GPT-3 model to create clones that mimic the user's speech patterns. The input is a model that represents the user's speech patterns, and the output is the generated clone.

[1783] Step 4:

[1784] The server updates the clone's speech patterns through chats between the generated clone and the user. Specifically, it collects chat data between the user and the clone and feeds it back into the model. The input is the chat data between the user and the clone, and the output is a model that represents the updated speech patterns.

[1785] Step 5:

[1786] The server generates the clone's response using a generative AI model. Specifically, it generates a prompt sentence and inputs it into the GPT-3 model to generate the clone's response. The input is the prompt sentence, and the output is the clone's response.

[1787] Step 6:

[1788] The server sends the clone's response to the user or a third party. Specifically, it sends the response via a chat application or email. The input is the clone's response, and the output is the message sent to the user or a third party.

[1789] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1790] "Example 1"

[1791] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system learns a user's speech patterns and generates a clone of the user based on the learned speech patterns. The clone's speech patterns are updated through chats between the generated clone and the user. Furthermore, this system includes an emotion engine that recognizes the user's emotions.

[1792] "Example 2"

[1793] The emotion engine recognizes emotions from the user's words and actions. For example, if a user says, "Today was fun," the emotion engine recognizes from this statement that the user is feeling happy. This recognition result is used to update the clone's speech patterns.

[1794] "Example 3"

[1795] Specifically, if the emotion engine recognizes joy, the clone is updated to make similar statements in situations that make the user feel joy. For example, the clone will say, "Today was fun." This allows the clone to make statements that reflect the user's emotions.

[1796] "Example 4"

[1797] The emotion engine is also used by the clone to adjust its conversation with third parties. For example, if the emotion engine recognizes that the user is feeling angry, the clone will use that information to appropriately adjust its conversation with third parties. Specifically, in situations where the user feels angry, the clone will make statements that express the user's anger.

[1798] The processing flow of each embodiment will be described below.

[1799] "Example 1"

[1800] Step 1: The system initiates a chat with the user through a messaging app.

[1801] Step 2: The system learns the user's speech patterns through chat.

[1802] Step 3: Based on the learned speech patterns, the system generates a clone of the user.

[1803] Step 4: Through chat between the generated clone and the user, the system updates the clone's speech patterns.

[1804] Step 5: The system activates the emotion engine to recognize the user's emotions.

[1805] "Example 2"

[1806] Step 1: The emotion engine recognizes emotions from the user's statements and actions.

[1807] Step 2: The recognized emotions are used to update the clone's speech patterns.

[1808] "Example 3"

[1809] Step 1: When the emotion engine recognizes a particular emotion, the clone's speech pattern is updated to reflect that emotion.

[1810] Step 2: Based on the updated speech patterns, the clone makes speech that expresses similar emotions to the user.

[1811] "Example 4"

[1812] Step 1: The emotion engine recognizes the user's emotion.

[1813] Step 2: Based on the perceived emotion, the clone adjusts the conversation with the third party.

[1814] Step 3: Specifically, the clone makes statements that reflect the emotions the user might be feeling.

[1815] Example 1

[1816] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1817] Conventional chat systems can learn a user's speech patterns and generate a user clone, but they lack the ability to recognize the user's emotions and generate responses accordingly. This makes it difficult to communicate in a way that is sensitive to the user's emotions, limiting the user experience.

[1818] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for collecting user utterance data, means for preprocessing the collected utterance data, means for learning utterance patterns using the preprocessed utterance data, means for generating a user clone based on the learned utterance patterns, means for updating the clone's utterance patterns through chat between the generated clone and the user, means for recognizing emotions from the user's utterances, and means for generating responses according to the recognized emotions. This makes it possible to learn the user's utterance patterns and generate responses according to the emotions.

[1819] The "means of collecting user comment data" refers to a function that saves comments made by users through the messenger app as a log and sends them to a server.

[1820] The "means for preprocessing collected utterance data" is a function that converts collected utterance data into a format that is easy to analyze by processing the data such as tokenizing, removing stop words, and stemming.

[1821] The "means for learning utterance patterns using preprocessed utterance data" is a function that uses a machine learning algorithm based on preprocessed data to analyze user utterance patterns and train a model.

[1822] The "means for generating a clone of a user based on learned speech patterns" is a function for creating a virtual agent that mimics the speech patterns of a user based on a learned model.

[1823] The "means for updating the clone's speech patterns through chat between the generated clone and the user" is a function that continuously learns and updates the clone's speech patterns using newly collected speech data through chat between the clone and the user.

[1824] The "means for recognizing emotions from user utterances" is a function that analyzes the content of a user's utterances and identifies the user's emotions using an emotion engine.

[1825] The "means for generating a response according to the recognized emotion" is a function for generating an appropriate response based on the user's emotion identified by the emotion engine.

[1826] This invention is a system that learns a user's speech patterns and generates a clone of the user based on the learning results. Furthermore, the generated clone updates its speech patterns through chats with the user, recognizes the user's emotions using an emotion engine, and generates responses accordingly.

[1827] The server uses a messenger app to collect user speech data. All speech made by users within the app is saved as a log and sent to the server. The collected speech data is preprocessed using natural language processing libraries such as Python's NLTK and spaCy. Preprocessing includes tokenizing the text data, removing stop words, and stemming.

[1828] The preprocessed speech data is used to learn speech patterns using machine learning frameworks such as TensorFlow and PyTorch. Specifically, a recurrent neural network (RNN) or transformer model is constructed, the user's speech patterns are analyzed, and the model is trained. A clone of the user is generated based on the learned model. The clone is a virtual agent that mimics the user's speech patterns.

[1829] The device then starts a chat between the user and the clone. When the user speaks through a messaging app, the clone responds accordingly. For example, if the user says "Good morning," the clone responds with "Good morning!"

[1830] Newly collected speech data from chats with the clone is sent to the server and used to update the clone's speech patterns, allowing the clone to reflect the user's latest speech patterns.

[1831] Furthermore, the device uses an emotion engine to recognize emotions from the user's speech. The emotion engine uses Microsoft Azure's Text Analytics API and IBM Watson's Tone Analyzer, among others. Once the emotion engine recognizes the user's emotion, it generates an appropriate response based on that emotion. For example, if the user says, "I'm tired today," the emotion engine will recognize the user's sense of fatigue and respond, "Thank you for your hard work. Please take a good rest today."

[1832] Example prompt sentence:

[1833] User: Good morning

[1834] Clone: ​​Good morning!

[1835] User: I'm tired today

[1836] Clone: ​​Thank you for your hard work. Please get some rest today.

[1837] In this way, the server, terminal, and user work together to learn the user's speech patterns, generate clones, and respond according to their emotions.

[1838] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1839] Step 1: Collect user utterance data

[1840] The server collects user speech data through the messenger app. When a user speaks within the app, the speech is saved as a log and sent to the server. The input is the user's speech text, and the output is the speech log saved on the server. Specifically, when a user says "Good morning," the text data is sent to the server.

[1841] Step 2: Preprocessing the utterance data

[1842] The server preprocesses the collected utterance data. The input is the collected utterance data, and the output is preprocessed text data. Specifically, it uses Python's NLTK and spaCy to tokenize the text data, remove stop words, and perform stemming. For example, the utterance "Good morning" is tokenized and unnecessary characters are removed.

[1843] Step 3: Learning speech patterns

[1844] The server uses preprocessed utterance data to learn utterance patterns. The input is preprocessed text data, and the output is a trained model. Specifically, a recurrent neural network (RNN) or transformer model is built using TensorFlow or PyTorch, and the user's utterance patterns are analyzed to train the model. For example, the utterance pattern "Good morning" is learned by the model.

[1845] Step 4: Creating a clone

[1846] The server generates a user clone based on the trained model. The input is the trained model, and the output is the user clone. The clone is a virtual agent that imitates the user's speech patterns. For example, a clone that says "Good morning" is generated.

[1847] Step 5: Chat with the clone

[1848] The device starts a chat between the generated clone and the user. The input is the user's speech, and the output is the clone's response. When the user speaks through a messenger app, the clone responds accordingly. For example, if the user says "Good morning," the clone responds "Good morning!"

[1849] Step 6: Update the clone's speech patterns

[1850] The server updates the clone's speech patterns using newly collected speech data through chats with the clone. The input is the newly collected speech data, and the output is the updated model. This allows the clone to reflect the user's latest speech patterns. For example, when a user says, "I'm tired today," that data is sent to the server, and the clone's speech patterns are updated.

[1851] Step 7: Emotion recognition and response generation by the emotion engine

[1852] The device uses an emotion engine to recognize emotions from what the user says. The input is what the user says, and the output is the recognized emotion and an appropriate response. Specifically, it uses Microsoft Azure's Text Analytics API and IBM Watson's Tone Analyzer. Once the emotion engine recognizes the user's emotion, it generates an appropriate response based on that emotion. For example, if the user says, "I'm tired today," the emotion engine will recognize the user's sense of fatigue and respond, "Thank you for your hard work. Please take it easy and rest today."

[1853] (Application example 1)

[1854] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1855] Conventional user interfaces and content recommendation systems have had difficulty fully understanding a user's speech patterns and emotions and providing individually optimized content. Furthermore, there were no systems that could converse with a third party on the user's behalf, which placed a heavy burden on the user.

[1856] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for learning a user's speech patterns, means for generating a user clone based on the learned speech patterns, means for updating the clone's speech patterns through chat between the generated clone and the user, means for recognizing the user's emotions, and means for recommending appropriate content based on the recognized emotions. This makes it possible to understand the user's speech patterns and emotions and provide individually optimized content. It is also possible to have a conversation with a third party on behalf of the user.

[1857] "User speech patterns" refers to the tendencies and characteristics of speech made by users through messenger apps and chat systems.

[1858] "Means of learning" refers to algorithms and systems that analyze users' speech patterns and accumulate their characteristics as data.

[1859] A "clone" is a virtual entity that is generated based on a user's speech patterns and imitates the same speech and behavior as the user.

[1860] "Means for updating the clone's speech patterns through chat" refers to a function that enables the clone to learn new speech patterns through dialogue with the user and to modify and update its own speech patterns.

[1861] "Means for recognizing emotions" refers to algorithms or systems that analyze emotions from a user's words and actions and identify their emotional state.

[1862] "Means for recommending appropriate content" refers to a function for selecting and providing the most suitable content (movies, music, articles, etc.) to a user based on the user's recognized emotions.

[1863] As an embodiment of the present invention, a system is constructed that learns user speech patterns, generates clones, recognizes emotions, and recommends appropriate content. A specific embodiment of this system is shown below.

[1864] System Configuration

[1865] The system consists of the following main components:

[1866] 1. User device: A device such as a smartphone or computer that users use to input their comments.

[1867] 2. Server: A central processing unit for learning user speech patterns, generating clones, recognizing emotions, and recommending content.

[1868] 3. Emotion Recognition Engine: Software that uses emotion analysis tools such as TextBlob to recognize emotions from user speech.

[1869] 4. Generative AI model: An algorithm that uses OpenAI's API to generate clone responses based on the user's speech patterns.

[1870] Program processing

[1871] The server performs the following process.

[1872] 1. A means of learning user speech patterns: Analyzes speech sent from the user's device and stores the speech patterns in a database. This allows us to understand the user's speech tendencies.

[1873] 2. Method for generating clones: Based on the learned speech patterns, a generative AI model is used to generate a clone of the user, which can make similar speech to the user.

[1874] 3. A method for updating the clone's speech patterns: Through chats between the clone and the user, new speech patterns are learned and the clone's speech patterns are updated.

[1875] 4. Means of Emotion Recognition: An emotion recognition engine is used to analyze emotions from the user's speech and identify their emotional state.

[1876] 5. Means of recommending appropriate content: Based on the recognized emotions, the most suitable content (movies, music, articles, etc.) for the user is selected and provided to the user's device.

[1877] Hardware and software used

[1878] Hardware: smartphones, PCs, servers

[1879] Software: Python, OpenAI API, TextBlob

[1880] Specific examples

[1881] If the user says "I'm so happy today!", the following happens:

[1882] 1. A message is sent from the user's device to the server.

[1883] 2. The server learns speech patterns and stores them in a database.

[1884] 3. Using a generative AI model, the clone responds, "That's great!"

[1885] 4. The emotion recognition engine analyzes the user's speech and recognizes positive emotions.

[1886] 5. The server recommends content based on positive emotions, such as "Here is a happy movie for you!"

[1887] Prompt Sentence Examples

[1888] User says: I'm so happy today!

[1889] AI Clone responds:

[1890] In this way, a system is realized that understands the user's speech patterns and emotions and provides individually optimized content.

[1891] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1892] Step 1:

[1893] A message is sent from the user terminal to the server.

[1894] Input: User utterance (e.g., "I'm so happy today!")

[1895] Processing: The user terminal sends the utterance as text data to the server.

[1896] Output: Speech data arrives at the server.

[1897] Step 2:

[1898] The server learns speech patterns and stores them in a database.

[1899] Input: User's speech data

[1900] Processing: The server analyzes the speech data and extracts speech patterns. The extracted patterns are stored in a database.

[1901] Output: Updated utterance pattern database

[1902] Step 3:

[1903] Using a generative AI model, the clone generates a response.

[1904] Input: User utterance data, utterance pattern database

[1905] Processing: The server uses the OpenAI API to generate a clone's response based on the user's utterance. The server inputs the prompt sentence into the generative AI model and obtains the response.

[1906] Output: The clone's response (e.g., "That's great!")

[1907] Step 4:

[1908] The clone's response is sent to the user terminal.

[1909] Input: Clone response

[1910] Processing: The server sends the generated clone response to the user terminal.

[1911] Output: The clone's response displayed on the user's terminal

[1912] Step 5:

[1913] The emotion recognition engine analyzes the user's speech and recognizes their emotions.

[1914] Input: User's speech data

[1915] Processing: The server uses TextBlob to parse the sentiment from the user's speech and identify positive, negative, and neutral sentiment states.

[1916] Output: Recognized emotion data (e.g., positive)

[1917] Step 6:

[1918] The server recommends appropriate content based on the recognized emotions.

[1919] Input: Recognized emotion data

[1920] Processing: The server selects the most suitable content for the user based on the emotion data. For example, if the emotion is positive, it will select content such as "Here is a happy movie for you!"

[1921] Output: Recommended content

[1922] Step 7:

[1923] The recommended content is transmitted to the user terminal.

[1924] Input: Recommended content

[1925] Processing: The server sends the recommended content to the user terminal.

[1926] Output: Recommended content displayed on the user's device

[1927] In this way, a system is realized that understands the user's speech patterns and emotions and provides individually optimized content.

[1928] Example 2

[1929] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1930] Conventional systems were able to learn the user's speech patterns and generate clones, but they were unable to recognize the user's emotions and dynamically update the clone's speech patterns based on those emotions. As a result, conversations with the user were one-sided, making it difficult to have natural dialogue. Furthermore, the inability to respond to the user's emotions led to a poor user experience.

[1931] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server include...

Claims

1. A learning means for learning a user's speech pattern using a generative AI model; a generating means for generating a clone of the user based on the learned utterance pattern; a means for updating the utterance pattern through a chat between the generated clone and the user; an emotion engine that recognizes the emotion of the user; A generating means for generating a response of the clone according to the emotion by inputting a specific prompt sentence into the generating AI model using the generating AI model; A means for using the generated response of the clone in a conversation with a third party; A system including:

2. The system according to claim 1 , wherein the generating means generates the clones based on the learned utterance patterns using a trained generative AI model.

3. The system of claim 1 , further comprising: means for training the generative AI model based on the updated utterance patterns.

Images