System

Abstract

A system is provided.SOLUTION: A system, comprising: means for acquiring a voice of an operator; means for generating an animation character based on the voice; means for synchronizing the generated animation character and the voice of the operator in real time; means for transmitting the synchronized voice and character to a user terminal; and means for a user to select an operator.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In traditional call centers, customers often feel uneasy because they cannot see the operator's face, which makes the explanation seem mechanical. There is also a need to provide the same sense of security as face-to-face contact for elderly users and those who feel uneasy about not being able to see the operator's face. Furthermore, there is a need for technology that can accommodate diversifying work styles and enable flexible work styles, such as working from home in call centers. [Means for solving the problem]

[0005] The present invention provides a system including: a means for acquiring an operator's voice; a means for generating an animated character based on the voice; a means for synchronizing the generated animated character and the operator's voice in real time; a means for transmitting the synchronized voice and character to a user terminal; and a means for a user to select an operator. Furthermore, by including a means for storing operator information in a database and a means for transmitting a list of waiting operators to a client based on the information, the user can select a desired operator from among available operators. Furthermore, by including a means for recording user interactions and a means for collecting feedback based on the recorded interactions, the quality of service can be improved. This system reduces the anxiety of not being able to see the operator's face and enables a service that can convey emotions in real time.

[0006] "Operator voice" refers to voice data uttered by an operator who responds to calls at a call center or the like.

[0007] An "animated character" is a two-dimensional or three-dimensional visual avatar that is generated based on the operator's voice.

[0008] "Synchronization" is the process of controlling the operator's voice and the animated character's movements so that they match in real time.

[0009] "User terminal" refers to a device, such as a smartphone or computer, used to connect to a call center and receive assistance.

[0010] The "means for selecting an operator" refers to an interface or operation method that allows the user to select a desired operator from a list.

[0011] The "list of waiting operators" is a list of operators who are currently available to respond to a call, and is presented to the user in a selectable state.

[0012] "Means for recording interactions" is a mechanism for saving interactions between users and operators for later analysis and feedback.

[0013] "Means for collecting feedback" refers to a system for collecting satisfaction and opinions from users after the service has ended. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The system according to the present invention synchronizes the voice of an operator with an animated character in real time, providing users with a more face-to-face interaction. This system can be implemented through the following steps:

[0036] server

[0037] 1. Register operator information

[0038] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[0039] The server sends a request to the image generation AI using the registered voice sample to generate an animated character of the operator.

[0040] The generated animated character is again saved in the database.

[0041] 2. Waiting for client connections

[0042] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[0043] The server maintains and updates a list of available operators.

[0044] 3. Operator selection and synchronization

[0045] When the server receives a request from a client, it sends a list of waiting operators to the client.

[0046] Once the client selects the desired operator, the information is returned to the server, which synchronizes the voice and animated character of the selected operator in real time.

[0047] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animated character.

[0048] 4. Sending voice and characters

[0049] The server transmits synchronized audio and animated characters to the client in real time.

[0050] Terminal (client)

[0051] 1. Sending a connection request

[0052] Users use their devices to send connection requests to a call center, which can be made by phone or through a dedicated application.

[0053] 2. Operator Selection

[0054] The list of waiting operators sent from the server is displayed on the terminal.

[0055] The user selects a desired animated character from the list and transmits the selection information to the server.

[0056] 3. Real-time synchronization

[0057] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[0058] 4. Recording interactions

[0059] After the interaction, the user's interaction is recorded and used to collect feedback and satisfaction.

[0060] Specific examples

[0061] Example of a system where Operator A is registered

[0062] Registering operator information

[0063] 1. The server registers Operator A's name, profile picture, and voice sample.

[0064] 2. The server uses operator A's voice sample to request the image generation AI to generate an animated character.

[0065] 3. The generated animated character is saved in the database.

[0066] User connects to call center

[0067] 1. A user connects to a call center using a phone or application.

[0068] 2. The server sends a list of waiting operators to the user terminal.

[0069] 3. The user selects an animated character for Operator A and sends that information back to the server.

[0070] Real-time synchronization begins

[0071] 1. The server captures operator A's voice in real time and synchronizes it with the movement data of the animation character that corresponds to that voice.

[0072] 2. Synchronized audio and animated characters are transmitted to the user's device in real time.

[0073] 3. Users can receive service as if they were face-to-face with Operator A through the animated character.

[0074] This system allows users to enjoy a more emotional and face-to-face service compared to traditional call center services. Furthermore, by recording interactions and collecting feedback, it is possible to improve the quality of service.

[0075] The processing flow will be explained below.

[0076] Server Processing Steps

[0077] Step 1:

[0078] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[0079] Step 2:

[0080] The server sends the voice samples to an image generation AI to generate an animated character for the operator.

[0081] Step 3:

[0082] The server stores the generated animated characters in a database.

[0083] Step 4:

[0084] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[0085] Step 5:

[0086] The server maintains an updated list of available operators.

[0087] Step 6:

[0088] The server receives a connection request from a client and sends a list of waiting operators to the client.

[0089] Step 7:

[0090] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[0091] Step 8:

[0092] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animation character.

[0093] Step 9:

[0094] The server transmits synchronized audio and animated characters to the client in real time.

[0095] Terminal (client) processing steps

[0096] Step 1:

[0097] The user uses a terminal to send a connection request to the call center, which can be made over the phone or through an application.

[0098] Step 2:

[0099] The list of waiting operators sent from the server is displayed on the terminal.

[0100] Step 3:

[0101] The user selects a desired animated character from the list and transmits the selection information to the server.

[0102] Step 4:

[0103] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[0104] Step 5:

[0105] After the interaction, the user's interaction is recorded and used to collect feedback and satisfaction.

[0106] User processing steps

[0107] Step 1:

[0108] The user connects to the call center by phone or using an application.

[0109] Step 2:

[0110] After connecting, check the list of waiting operators and select the character you want.

[0111] Step 3:

[0112] The selected animated character will synchronize with the audio in real time and begin interacting.

[0113] Step 4:

[0114] After the interaction, use the feedback form to provide your satisfaction and opinions.

[0115] Example 1

[0116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] In conventional call center systems, users communicate with operators only through voice, which means there is a lack of visual information, making it difficult to convey emotions and facial expressions compared to face-to-face interactions. Furthermore, the limited options for operators can lead to lower user satisfaction. Furthermore, the lack of interaction recording and feedback collection functions poses a challenge, resulting in a lack of data to improve service quality.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0119] In this invention, the server includes a means for acquiring the voice of an operator, a means for generating a prompt sentence for a generative AI model based on the voice and generating an animated character, a means for synchronizing the generated animated character with the voice of the operator in real time, a means for transmitting the synchronized voice and character to the client, and a means for the user to select an animated character for the operator. This allows users to enjoy a visually rich, face-to-face interaction, and the increased choice of operators increases satisfaction. Furthermore, by recording interactions and collecting feedback, the quality of service can be continuously improved.

[0120] "Operator" means a person who provides services and support to users via telecommunications.

[0121] "Voice" refers to data of the voice or speech produced by the operator.

[0122] A "generative AI model" is an artificial intelligence algorithm or program that generates new data based on input data. Specifically, it refers to a model for generating animated characters from voice or text.

[0123] A "prompt" is text that is input to a generative AI model and contains instructions and conditions for obtaining a specific output.

[0124] An "animated character" is a virtual figure or character generated based on the operator's voice and movements.

[0125] "Real-time" means a state in which delays are extremely small and immediate responses are possible.

[0126] "Synchronization" means moving or matching multiple pieces of data or actions simultaneously.

[0127] A "client" refers to a terminal or software for using the system, and is connected to the server primarily to process user requests.

[0128] A "database" is a collection of information organized to efficiently store, retrieve, and update data.

[0129] "Interaction" is the process of exchange or operation between a user and a system or operator.

[0130] "Feedback" refers to the evaluations and opinions provided by users regarding the quality and usability of the Service, and includes the process of improving the Service based on the collected information.

[0131] The system according to the present invention synchronizes the voice of an operator with an animated character in real time, providing users with a service that is almost like a face-to-face interaction. This system is configured as follows.

[0132] server

[0133] The server registers the operator's basic information (name, profile photo, voice sample) in a database. After registration, the voice sample is used to generate a prompt for a generative AI model (e.g., DALL-E or Stable Diffusion) to generate an animated character. Specifically, an example of a prompt might be, "Based on this voice sample, please generate a real-time animated character for Operator A." The generated animated character is saved in the database.

[0134] The server waits for a connection request from the user. Once a connection is made, it sends a list of available operators to the client. Once the user selects an operator, it uses this information to synchronize the voice and animated characters in real time and transmits them to the client using a low-latency protocol (e.g., WebRTC).

[0135] Terminal (client)

[0136] The user uses their device to send a connection request to the call center through a dedicated application or browser. Once connected, a list of available operators sent from the server is displayed, and the user selects the desired operator. The selection information is sent back to the server, and real-time voice and animated characters are displayed on the device. The necessary data is received via a low-latency protocol such as WebRTC and rendered in real time.

[0137] User

[0138] Users can easily connect to the call center using a dedicated application or browser and select an operator. After the call is completed, the interaction is recorded and users can provide feedback to improve the quality of the service. This interaction and feedback are stored in a database and used to improve the service.

[0139] Specific examples

[0140] An example will be explained in which operator A is newly registered.

[0141] 1. Register operator information

[0142] The server registers Operator A's name, profile picture and voice sample in a database.

[0143] The server uses the voice sample to generate a prompt for the generation AI (e.g., Stable Diffusion), specifically, the prompt sentence "Based on this voice sample, please generate a real-time animated character of Operator A."

[0144] The generated animated characters are stored in a database.

[0145] 2. The user connects to the call center

[0146] Users use a dedicated application to connect to the call center by sending an HTTP request.

[0147] The server sends a list of waiting operators to the user terminal.

[0148] The user selects the desired operator and returns the selection information to the server.

[0149] 3. Start syncing in real time

[0150] The server captures the audio data in real time and synchronizes it with the animated character using an AI animation engine (e.g., Live2D).

[0151] The server sends the synchronized data to the client using WebRTC.

[0152] The user can view the transmitted audio and animated characters in real time using a dedicated application.

[0153] In this way, the system of the present invention allows users to enjoy visually rich, face-to-face interaction, improving the quality of service.

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Step 1: Register operator information

[0156] The server registers basic information of the operator (name, profile photo, voice sample) in the database. As input, it receives the operator's name, profile photo, and voice sample. In concrete terms, it collects the voice sample and stores it in the "operators" table of the database along with the name and photo. The output is the operator information registered in the database.

[0157] Step 2: Sending a request to the image generation AI

[0158] The server uses the registered voice sample to generate a prompt for a generative AI model (e.g., DALL-E or Stable Diffusion) and generates an animated character. The input in this step is the voice sample, and an example prompt is "Please generate a real-time animated character for Operator A based on this voice sample." Specifically, the server sends a prompt to the generative AI model to generate an animated character. The output is data for the generated animated character, which is stored in a database.

[0159] Step 3: Waiting for client connections

[0160] The server waits for connection requests from users. The input includes user connection requests, which may come over telephone lines or the Internet. Specific operations include listening on a TCP / IP port or using a telephony API. The output is a list of connected clients.

[0161] Step 4: Submit the Operator List

[0162] When the server receives a connection request, it sends a list of waiting operators to the client. The input in this step is the waiting operator information retrieved from the database, and this information is sent to the client in JSON format. The output is the operator list displayed on the user's terminal.

[0163] Step 5: Receive operator selection information

[0164] When the user selects the desired operator, the selection information is sent back to the server. The input of this step is the operator information selected by the user, specifically, the selection information is sent in JSON format. The output is the selection information received by the server.

[0165] Step 6: Synchronizing the audio with the animated character

[0166] The server captures the selected operator's voice in real time and synchronizes it with the corresponding animated character's movement data. The input in this step is real-time voice data and the generated animated character's movement data. Specifically, the voice data is synchronized using a streaming protocol (e.g., WebRTC), and the movement data is synchronized using an AI animation engine (e.g., Live2D). The output is the synchronized voice and animated character data.

[0167] Step 7: Sending voice and animated characters

[0168] The server sends synchronized audio and animated character data to the client in real time. The input in this step is the synchronized audio and animated character data, specifically, the transmission uses WebSocket or other low-latency protocols. The output is the data sent to the client.

[0169] Step 8: Sending a Connection Request

[0170] The user uses a terminal to send a connection request to the call center through a dedicated application or browser. The input to this step is the user's connection request, specifically, using an HTTP request or WebSocket. The output is the connection request sent to the server.

[0171] Step 9: Select an operator

[0172] The list of waiting operators sent from the server is displayed on the terminal. The user selects the desired operator from the list and sends the selection information to the server. The input in this step is the list of waiting operators, and the output is the operator information selected by the user.

[0173] Step 10: Viewing Real-Time Sync

[0174] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal. The input is synchronized data transmitted from the server, specifically received via a low-latency protocol (e.g., WebRTC). The output is the animated character and voice displayed on the user's terminal.

[0175] Step 11: Record the interaction

[0176] Once the call is over, the user interaction is recorded. The input to this step is the data of the user interaction, specifically stored in the "interactions" table in the database. The output is the recorded interaction data.

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] Conventional call center systems rely mainly on voice communication, and lack visual interaction with users, resulting in lower user satisfaction compared to face-to-face support. Furthermore, there are also challenges such as the difficulty of providing prompt and friendly responses in emergencies and security-related inquiries.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the voice of the operator in real time, means for transmitting the synchronized voice and character to a user terminal, means for a user to select an operator, and means for applying the character display and voice synchronization to emergency calls and inquiries. This allows users to communicate with operators in real time in a visually friendly manner. This also allows for a rapid response in security-related emergencies.

[0182] "Operator" refers to a staff member who responds to user inquiries.

[0183] "Means for capturing audio" refers to a device or software for capturing the operator's speech in real time.

[0184] "Animated Character" refers to a dynamic character that is generated based on the operator's voice and visually displayed to the user.

[0185] "Real-time synchronization means" means any device or software that synchronizes the voice of an operator with the movements of an animated character.

[0186] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0187] The term "means for a user to select an operator" refers to an interface or function that allows a user to select any operator from a plurality of operators.

[0188] "Applicable emergency communication and inquiry response measures" refers to the functions or processes that ensure efficient response to security-related emergencies and other inquiries.

[0189] The system for implementing this invention is composed of a server, a client terminal, and a user as its main components. The overall flow is that the server registers operator information, and based on that information, synchronizes the operator's voice with an animated character in real time and transmits it to the user terminal.

[0190] Server Configuration

[0191] The server consists of the following hardware and software:

[0192] Hardware: Server equipped with high-performance CPU, memory, storage, etc. (e.g. Dell PowerEdge)

[0193] Software: Python and Django for server-side processing, SQL for database management, TensorFlow and custom AI models for image generation and audio synchronization

[0194] function:

[0195] 1. Operator information registration:

[0196] The operator's name, profile picture and voice sample are entered into a database.

[0197] Using a voice sample, we request the image generation AI to generate an animated character.

[0198] The generated animated character is saved in a database.

[0199] 2. Waiting for client connections:

[0200] Waits for call center connection requests via telephone lines or the Internet.

[0201] 3. Operator selection and synchronization:

[0202] Receives requests from clients and sends a list of waiting operators to the user's terminal.

[0203] Real-time synchronization of voice and animated characters for selected operators.

[0204] 4. Voice and character transmission:

[0205] Synchronized voice and animated characters are transmitted to a user terminal in real time.

[0206] Client terminal

[0207] Client devices are primarily expected to be smartphones. Applications are developed and operated in the following environments:

[0208] Hardware: Android smartphone or iPhone

[0209] Software: Kotlin (for Android), Swift (for iOS), Unity (for animation display)

[0210] function:

[0211] 1. Send a connection request:

[0212] The user uses the application on the terminal to send a connection request to the call center.

[0213] 2. Operator Selection:

[0214] The list of waiting operators sent from the server is displayed on the terminal.

[0215] Provides an interface for users to select their preferred operator.

[0216] 3. Real-time Sync:

[0217] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[0218] 4. Recording interactions:

[0219] After the interaction, the user's interaction is recorded to collect feedback and satisfaction.

[0220] Specific examples

[0221] When a user detects an abnormality in their home security system during an emergency, they connect to the "Security Assistant" through the app. A list of operators is displayed, and the user can select one. The selected operator's animated character will appear in real time, and emergency response instructions will be given through synchronized voice.

[0222] Prompt Sentence Examples

[0223] Examples of prompts include:

[0224] "Please register your operator information. The operator's name is "Taro", the profile picture is the attached image, and the voice sample is the attached audio file. Please generate an animated character based on this information."

[0225] This configuration allows users to communicate visually with operators in real time and receive prompt and accurate responses, and also enables rapid response in security-related emergencies.

[0226] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0227] Step 1:

[0228] The server registers the operator's name, profile picture, and voice sample in the database. The input of this step is the operator's basic information, and the output is the operator's information stored in the database. Data processing involves analyzing the input voice sample, converting it into an appropriate format, and saving it.

[0229] Step 2:

[0230] The server uses the registered voice sample to request the image generation AI to generate an animated character. The input of this step is the voice sample, and the output is the generated animated character. Data calculation involves extracting voice features and inputting them into the image generation AI to generate the animated character.

[0231] Step 3:

[0232] The server saves the generated animated character in a database. The input of this step is the generated animated character, and the output is the character data saved in the database. Data processing involves converting the character data into an appropriate format and saving it.

[0233] Step 4:

[0234] The user uses the terminal to send a connection request to the call center. The input of this step is the user's connection request, and the output is the connection request information sent to the server. The specific operation is that the user opens the application and presses the call center connection button.

[0235] Step 5:

[0236] The server receives a call center connection request and sends a list of waiting operators to the user's terminal. The input of this step is the user's connection request, and the output is the list of waiting operators. Data processing involves converting the waiting operator information into a list format and sending it.

[0237] Step 6:

[0238] The user selects the desired operator from the list and sends the selection information to the server. The input of this step is the operator selection information, and the output is the selection information sent to the server. The specific operation is that the user taps on the terminal screen to select the operator.

[0239] Step 7:

[0240] The server synchronizes the voice of the selected operator with the animation character in real time. The input of this step is the operator's voice, and the output is the synchronized animation character. The data calculation is to analyze the voice data in real time and generate the animation movement accordingly.

[0241] Step 8:

[0242] The server transmits the synchronized voice and animated characters to the user terminal in real time. The input of this step is the synchronized voice and characters, and the output is the data transmitted to the user terminal. The data processing is to perform real-time data streaming.

[0243] Step 9:

[0244] The user interacts with the operator through an animated character displayed in real time on the terminal. The input of this step is the voice and character sent to the terminal, and the output is the user's interaction experience. The specific operation is for the user to continue the conversation with the operator while looking at the terminal screen.

[0245] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0246] The present invention relates to a system that combines real-time synchronization of an operator's voice with an animated character, and an emotion engine for recognizing the user's emotions. The system is implemented through the following steps:

[0247] server

[0248] 1. Register operator information

[0249] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[0250] The server sends a request to the image generation AI using the registered voice sample to generate an animated character of the operator.

[0251] The generated animated characters are stored in a database.

[0252] 2. Waiting for client connections

[0253] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[0254] The server maintains and updates a list of available operators.

[0255] 3. Operator selection and synchronization

[0256] The server receives a request from a client and sends a list of waiting operators to the client.

[0257] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[0258] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animated character.

[0259] 4. Emotion Recognition by Emotion Engine

[0260] The server uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[0261] The recognized emotion data is used to change the facial expressions and movements of the operator's animated character.

[0262] 5. Voice and character transmission

[0263] The server transmits synchronized audio and animated characters to the client in real time.

[0264] 6. Recording interaction and emotion data

[0265] The server records the user's interactions and the perceived emotion data and integrates it into subsequent feedback collection measures.

[0266] Terminal (client)

[0267] 1. Sending a connection request

[0268] Users use their devices to send connection requests to a call center, which can be made via telephone or a dedicated application.

[0269] 2. Operator Selection

[0270] The list of waiting operators sent from the server is displayed on the terminal.

[0271] The user selects a desired animated character from the list and transmits the selection information to the server.

[0272] 3. Real-time synchronization

[0273] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[0274] 4. Emotion-aware feedback

[0275] The terminal receives the user's emotion recognized by the emotion engine and provides an operator's response based on the emotion.

[0276] 5. Recording interactions

[0277] After the interaction, the user's interaction and emotional data is recorded and used to collect feedback and satisfaction.

[0278] User processing steps

[0279] 1. Connect to the call center

[0280] The user connects to the call center by phone or using an application.

[0281] 2. Confirm your operator selection

[0282] After connecting, check the list of waiting operators and select the character you want.

[0283] 3. Start of the conversation

[0284] The selected animated character will synchronize with the audio in real time and begin interacting.

[0285] 4. Providing emotional feedback

[0286] After the call is over, the emotional data recognized by the emotion engine is integrated into a feedback form to provide satisfaction and opinions.

[0287] Specific examples

[0288] Example of a system where Operator A is registered

[0289] Registering operator information

[0290] 1. The server registers Operator A's name, profile picture, and voice sample.

[0291] 2. The server uses operator A's voice sample to request the image generation AI to generate an animated character.

[0292] 3. The generated animated character is saved in the database.

[0293] User connects to call center

[0294] 1. A user connects to a call center using a phone or application.

[0295] 2. The server sends a list of waiting operators to the user terminal.

[0296] 3. The user selects an animated character for Operator A and sends that information back to the server.

[0297] Real-time synchronization begins

[0298] 1. The server captures operator A's voice in real time and synchronizes it with the corresponding animation character's movement data.

[0299] 2. Synchronized audio and animated characters are transmitted to the user's device in real time.

[0300] 3. The emotion engine recognizes the user's emotions and uses that data to change the facial expression of the operator's animated character.

[0301] 4. Users can receive service through the animated character as if they were face-to-face with Operator A.

[0302] This system allows users to enjoy a more emotional and face-to-face service compared to traditional call center services. Furthermore, by recording interactions and collecting feedback, it is possible to improve the quality of service.

[0303] The processing flow will be explained below.

[0304] Server Processing Steps

[0305] Step 1:

[0306] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[0307] Step 2:

[0308] The server sends the registered voice samples to an image generation AI to generate an animated character of the operator.

[0309] Step 3:

[0310] The server stores the generated animated characters in a database.

[0311] Step 4:

[0312] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[0313] Step 5:

[0314] The server maintains an updated list of available operators.

[0315] Step 6:

[0316] The server receives a connection request from a client and sends a list of waiting operators to the client.

[0317] Step 7:

[0318] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[0319] Step 8:

[0320] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animation character.

[0321] Step 9:

[0322] The server uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[0323] Step 10:

[0324] The server changes the facial expressions and movements of the operator's animated character based on the recognized emotional data.

[0325] Step 11:

[0326] The server transmits synchronized audio and animated characters to the client in real time.

[0327] Step 12:

[0328] The server records the user's interactions and the perceived emotion data and integrates it into subsequent feedback collection measures.

[0329] Terminal (client) processing steps

[0330] Step 1:

[0331] Users use their devices to send connection requests to a call center, which can be made via telephone or a dedicated application.

[0332] Step 2:

[0333] The list of waiting operators sent from the server is displayed on the terminal.

[0334] Step 3:

[0335] The user selects a desired animated character from the list and transmits the selection information to the server.

[0336] Step 4:

[0337] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[0338] Step 5:

[0339] The terminal receives the user's emotions recognized by the emotion engine and changes the facial expression of the operator's animated character in real time based on that information.

[0340] Step 6:

[0341] After answering the call, the terminal records the user's interaction and the recognized emotion data and transmits them to the server.

[0342] User processing steps

[0343] Step 1:

[0344] The user connects to the call center by phone or using an application.

[0345] Step 2:

[0346] After connecting, check the list of waiting operators and select the character you want.

[0347] Step 3:

[0348] The selected animated character will synchronize with the audio in real time and begin interacting.

[0349] Step 4:

[0350] Interact with an operator whose facial expressions change in real time thanks to an emotion engine.

[0351] Step 5:

[0352] After the call is over, the emotional data recognized by the emotion engine is integrated into a feedback form to provide satisfaction and opinions.

[0353] Through the above steps, the system can provide users with an emotional and face-to-face interaction, and the recording of interaction and emotional data provides valuable data for improving service quality.

[0354] Example 2

[0355] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] Conventional call center systems mainly rely on voice communication between operators and users, limiting interaction. Furthermore, they are unable to recognize users' emotions in real time and respond accordingly, resulting in a poor user experience. Furthermore, processes such as operator selection and interaction recording are inefficient, making it difficult to improve service quality. There is a need to solve these problems and provide responses that are more emotional and closer to face-to-face interactions.

[0357] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0358] In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the operator's voice in real time, means for transmitting the synchronized voice and character to a user terminal, means for the user to select an operator, means for recognizing the user's emotion, and means for changing the facial expressions and movements of the animated character based on the recognized emotion. This allows the user to receive emotional responses in real time, allowing for an interaction experience that is closer to face-to-face interaction. Furthermore, feedback can be collected based on interaction records and emotion data, making it possible to continuously improve the quality of service.

[0359] "Means for acquiring the operator's voice" refers to devices or software for capturing the voice data spoken by the operator in real time and inputting it into the system.

[0360] "Means for generating animated characters" means technologies and algorithms for generating animated characters that can move in real time based on operator voice and profile data.

[0361] "Means for synchronizing voice and characters in real time" refers to software or systems that allow for timely adjustment and reflection of the movements and expressions of animated characters based on the operator's voice data.

[0362] "Means for transmitting to the user terminal" refers to the communication technology and protocol for transmitting synchronized audio data and animation data from the server to the client terminal in real time.

[0363] "Means for the user to select an operator" refers to a GUI (graphical user interface) on the interface used by the user to select a desired operator from a plurality of operators.

[0364] "Means for recognizing user emotions" refers to an emotion recognition engine or algorithm that processes the user's voice data and facial expression data and analyzes their emotional state.

[0365] "Means for changing the facial expressions and movements of an animated character based on emotions" refers to a control system or algorithm that uses recognized user emotional data to change the facial expressions and movements of an animated character in real time.

[0366] "Means for storing operator information in a database" refers to the functions and systems for registering and storing basic information about operators (such as names, profile photos, and voice samples) in a database.

[0367] "Means for sending a list of waiting operators to a client" refers to a communication means or protocol for generating a list of waiting operators on the server side and sending it to the user's terminal.

[0368] "Means for recording interactions" refers to technologies and systems that record all voice interactions and actions between users and operators for later analysis and evaluation.

[0369] "Means for collecting feedback based on emotional data" refers to a method for analyzing emotional data collected during interactions with users and using the results to collect user satisfaction and areas for improvement through a feedback form, etc.

[0370] "Measures to improve service quality" refers to continuous improvement methods that analyze collected feedback data and use it to improve the way operators respond and the performance of the entire system.

[0371] MODE FOR CARRYING OUT THE INVENTION

[0372] This invention relates to a system that synchronizes voice and animation in real time and recognizes a user's emotions. This system is composed of a server, a terminal (client), and a user.

[0373] Basic Server Configuration

[0374] The server is the central component responsible for the main processing and implements the following functions:

[0375] 1. Register operator information

[0376] The server stores various information about the operator (name, profile photo, voice sample) in a database. An RDBMS such as MySQL is used here. Using Python's OpenAI API library, the voice sample is sent to a generative AI model (e.g., DALL-E) to generate an animated character. The generated character is then stored back in the database.

[0377] 2. Waiting for client connections

[0378] The server waits for and processes connection requests from clients using a communications platform such as Twilio.

[0379] 3. Operator selection and synchronization

[0380] Based on the client's request, the server sends a list of available operators to the client. When the user selects an operator, the server receives that information and synchronizes the operator's voice data with the animated character in real time. APIs such as Amazon Polly and Google Text-to-Speech are used for voice processing.

[0381] 4. Emotion Recognition by Emotion Engine

[0382] The server uses Azure Cognitive Services' Emotion API to recognize emotions from the user's voice and facial expressions, and the recognized emotional data is reflected in the facial expressions and movements of the animated character in real time.

[0383] 5. Voice and character transmission

[0384] The server sends synchronized audio and animated characters to the client using real-time communication such as WebSocket.

[0385] 6. Recording interaction and emotion data

[0386] The server records user interactions and emotional data and stores it in Amazon S3 or Google Cloud Storage, which helps gather subsequent feedback and improve the service.

[0387] Basic configuration of the terminal (client)

[0388] The client terminal is a device used by a user and has the following functions:

[0389] 1. Sending a connection request

[0390] Users use a dedicated smartphone app or web application to send a connection request to a server, which is established through a service such as Twilio.

[0391] 2. Operator Selection

[0392] The user checks the list of waiting operators sent from the server and selects the desired operator, and the selection information is sent to the server.

[0393] 3. Real-time synchronization

[0394] The client receives and displays the selected operator's voice and animated character in real time. The animated character moves in real time using the Unity engine or similar.

[0395] 4. Emotion-aware feedback

[0396] The client receives the user's emotion data recognized by the emotion engine and displays a response based on that data (such as changing the character's facial expression).

[0397] 5. Recording interactions

[0398] The client records the user's interaction and emotion data and sends it to the server, which uses the data to improve satisfaction and services.

[0399] Basic user actions

[0400] A user uses the system in the following steps:

[0401] 1. Connect to the call center

[0402] Users connect to the call center using a phone or an application, with the connection provided through a communications platform such as Twilio.

[0403] 2. Confirm your operator selection

[0404] The user checks the list of waiting operators presented by the server and selects a desired operator.

[0405] 3. Start of the conversation

[0406] The selected animated character will be synchronized with the voice in real time, allowing the user to receive a face-to-face response, and the emotion engine will recognize the user's emotions in real time, providing a more emotionally relevant response.

[0407] 4. Providing emotional feedback

[0408] After the call is completed, a feedback form is displayed based on the emotional data recognized by the emotion engine, allowing the user to provide their satisfaction and opinions.

[0409] Specific examples

[0410] Prompt Sentence Examples

[0411] Example: Use the voice sample of Operator A to generate an animated character using a generative AI model. Voice sample URL: http: / / example.com / audio_sample

[0412] Compared to traditional call centers, the system offers a more empathetic, face-to-face experience, and by recording interactions and collecting feedback, it can continuously improve service quality.

[0413] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0414] Program processing flow

[0415] server

[0416] Step 1: Register operator information

[0417] Input: Operator name, profile picture, voice sample

[0418] What it does: The server receives the operator's name, profile picture, and voice sample from the form and stores the data in a MySQL database.

[0419] Data processing: The server uses Python's OpenAI API library to send voice samples to DALL-E with prompt sentences to generate a corresponding animated character.

[0420] Output: The generated animated characters are saved back to the database.

[0421] Step 2: Waiting for client connections

[0422] Input: Connection request from the client

[0423] Specific operation: The server waits for a call center connection request from a client using a cloud communication platform such as Twilio.

[0424] Data Processing: Manage and update the list of available operators in real time.

[0425] Output: Generates the operator list upon receiving a request from the client.

[0426] Step 3: Select and sync operators

[0427] Input: Operator selection information from the user

[0428] Specific operation: Based on a request from a client, the server sends a list of waiting operators to the client.

[0429] Data processing: The voice of the selected operator is synchronized with the movement data of the corresponding animated character in real time. Amazon Polly is used for voice processing.

[0430] Output: Synchronized data is generated.

[0431] Step 4: Emotion Recognition with the Emotion Engine

[0432] Input: User's voice and facial expressions

[0433] Specific operation: The server uses the Emotion API of Azure Cognitive Services to recognize emotions from the user's voice and facial expressions.

[0434] Data processing: Collect recognized emotion data and update the facial expressions and movements of the animated character.

[0435] Output: Updated animated character data is generated.

[0436] Step 5: Submit your voice and character

[0437] Input: Synchronized audio and animated character data

[0438] Specific operation: The server transmits synchronized audio and animated characters to the client in real time using WebSocket.

[0439] Data processing: Convert the data into a WebSocket communication format.

[0440] Output: Generates data sent to the client.

[0441] Step 6: Recording interaction and emotion data

[0442] Input: User interaction data and recognized emotion data

[0443] Specific Operation: The server records the user's interactions and the recognized emotion data.

[0444] Data processing: The recorded data is stored on Amazon S3 or Google Cloud Storage.

[0445] Output: The saved data is generated.

[0446] Terminal (client)

[0447] Step 1: Sending a Connection Request

[0448] Input: A user requesting a connection

[0449] How it works: A user sends a connection request to a server using a dedicated smartphone app or web application. This connection is made through a service such as Twilio.

[0450] Data processing: Generate connection request data.

[0451] Output: Generates the connection request data sent to the server.

[0452] Step 2: Select an operator

[0453] Input: List of waiting operators from the server

[0454] Specific operation: The user checks the list of waiting operators sent from the server and selects the desired operator. The selection information is sent to the server.

[0455] Data processing: Generate user selection information.

[0456] Output: Selection information is generated that is sent to the server.

[0457] Step 3: Real-time sync

[0458] Input: Synchronized audio and animated character data sent from the server

[0459] Specific operation: The client receives and displays the selected operator's voice and animated character in real time. The animated character moves in real time using the Unity engine or similar.

[0460] Data processing: Converting received data into a playable format.

[0461] Output: Synchronized voice and character data is generated that is displayed to the user.

[0462] Step 4: Emotion recognition feedback

[0463] Input: Emotion data from the server

[0464] Specific operation: The client receives the user emotion data recognized by the emotion recognition engine and displays a response based on that data (such as changing the character's facial expression).

[0465] Data processing: Generate animation update data based on emotion data.

[0466] Output: Updated animated character data is generated that is displayed to the user.

[0467] Step 5: Record the interaction

[0468] Input: User interaction data and emotion data

[0469] Specific operation: After the interaction, the client records the user's interaction data and emotion data and sends them to the server.

[0470] Data processing: Generate recorded interaction data and emotion data.

[0471] Output: Interaction data and emotion data are generated and sent to the server.

[0472] User

[0473] Step 1: Connect to a call center

[0474] Input: A user requesting a connection

[0475] What it does: A user connects to a call center using a phone or a dedicated app, and the connection is made through a communications platform such as Twilio.

[0476] Data processing: Generate connection request data and send it to the server.

[0477] Output: The connection request data sent to the server is generated.

[0478] Step 2: Confirm operator selection

[0479] Input: List of waiting operators from the server

[0480] Specific operation: The user checks the list of waiting operators presented by the server and selects the desired operator.

[0481] Data processing: Generate user selection information and send it to the server.

[0482] Output: Selection information is generated that is sent to the server.

[0483] Step 3: Start the conversation

[0484] Input: Synchronized audio and animated character data sent from the server

[0485] Specific operation: An animated character selected by the user is synchronized with the audio in real time, and the user is treated as if they were face-to-face.

[0486] Data processing: Converting received audio and animation data into a playable format.

[0487] Output: Synchronized voice and character data is generated that is displayed to the user.

[0488] Step 4: Provide emotional feedback

[0489] Input: Emotion data from the server

[0490] Specific operation: Based on the emotion data recognized by the emotion engine, a feedback form is displayed on the user's device. The user provides their satisfaction and opinions.

[0491] Data processing: Generate user feedback data and send it to the server.

[0492] Output: Feedback data is generated that is sent to the server.

[0493] (Application example 2)

[0494] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0495] Conventional customer service systems have the problem that the operator's facial expressions and emotions are difficult to convey, making communication with the user feel mechanical and cold. Furthermore, the lack of a system that can flexibly respond to the user's choices and emotions makes it difficult to provide highly satisfying service. This is particularly true in brick-and-mortar stores, where it is difficult for operators in remote locations to provide effective customer service in real time.

[0496] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the voice of the operator in real time, means for transmitting the synchronized voice and character to a user terminal, means for the user to select an operator, means for recognizing the user's emotion, and means for changing the facial expression of the animated character based on the recognized emotion. This allows the user to enjoy an emotional response similar to that of a face-to-face interaction, thereby enabling highly satisfying customer service.

[0497] "Means for acquiring the operator's voice" means a device or software for receiving the voice signal emitted by the operator and recording and processing it as digital data.

[0498] "Means for generating animated characters" means algorithms or software for automatically creating 2D or 3D animated characters based on operator voice or other data.

[0499] "Means of real-time synchronization" refers to technology or systems that instantly match the operator's voice and the movements of the animated character, displaying them without any time lag.

[0500] The "means for transmitting to the user terminal" is a communication function for transmitting the generated animated character and the audio synchronized therewith to the user's device via a network.

[0501] "Means for a user to select an operator" refers to an interface or system that allows a user to select a specific operator from among multiple operators.

[0502] "Means for recognizing user emotions" refers to algorithms and implementations for analyzing and identifying the emotional state of a user from their facial expressions and voice.

[0503] "Means for changing the facial expression of an animated character" refers to a function or system for appropriately changing the facial expression or movement of an animated character in accordance with the recognized emotions of the user.

[0504] "Means for storing operator information in a database" refers to a database system that centrally manages and stores basic information about operators, voice samples, generated character information, etc.

[0505] The "means for transmitting a list of waiting operators to a client" refers to a function or technology for transmitting a list of currently available operators to a user's device.

[0506] The "means for generating data for simulating information about a product" is a simulation function for automatically generating data required for product explanations and demonstrations.

[0507] The "means for providing product information to users" refers to a display function or interface for effectively conveying the generated product data to users.

[0508] "Means for recording user interactions" refers to a system for saving the content of conversations and actions with users as digital data.

[0509] "Means of collecting feedback" are surveys and rating systems for collecting user satisfaction and opinions.

[0510] "Means for dynamically changing customer service content based on user interaction" refers to algorithms and technologies that adapt and change customer service content in real time according to the user's dialogue and behavior.

[0511] The present invention relates to a system for recognizing a user's emotions by synchronizing an operator's voice with an animated character in real time. A specific embodiment of this system will now be described.

[0512] server

[0513] The server first captures the operator's voice. Commonly used hardware includes a microphone to record the voice as digital data and a server computer to process it. The voice data is captured using digital signal processing (DSP) technology.

[0514] The server then generates an animated character based on the voice data. The software used is a generative AI model (e.g., OpenAI's DALL-E or MidJourney). It analyzes the voice sample and generates an animated character based on its characteristics. This generated character is then stored in a database.

[0515] The server analyzes the voice data and links it with the character's movement data to synchronize the generated animated character and the operator's voice in real time. This synchronization is achieved using real-time voice analysis software and motion capture technology.

[0516] The server has a communication function to transmit synchronized audio and animated characters to the user terminal, using the commonly used Internet Protocol (TCP / IP) as the communication protocol.

[0517] The server also recognizes the user's emotions and dynamically changes the facial expression of the animated character based on that data. This emotion recognition uses the Hugging Face transformers library and DeepFace. The server analyzes emotions from the user's voice and facial expressions, dynamically changing the character's facial expression accordingly.

[0518] In addition, the server stores operator information in a database and has the function of sending a list of waiting operators to the user's terminal. It also plays a role in improving the efficiency of customer service by generating data to simulate product information and providing it to the user.

[0519] Terminal (client)

[0520] The user terminals include smartphones and smart glasses, and connect to the call center using these terminals. The user can send a connection request through the application and check the list of waiting operators. This list is provided by the server, and the user can select the operator they want.

[0521] Once the selection is complete, the voice and animated character are synchronized from the server and displayed in real time on the user's device. The device is equipped with a camera and microphone to recognize the user's emotions. This emotional data is sent to the server, and the character's facial expression changes dynamically based on the content of the conversation.

[0522] User

[0523] Users interact with the virtual store clerk through the terminal, and the system also has the ability to record user interactions and collect feedback, which is used to improve subsequent services.

[0524] Specifically, when a user visits a store and launches the app on their smartphone, a list of available virtual clerks is displayed. The user selects a virtual clerk from the list and begins a conversation. The system then recognizes the user's emotions and changes the virtual clerk's response accordingly. The following is an example of a prompt sentence:

[0525] Example prompt sentence:

[0526] "The user launches the app and gets a list of available virtual sales associates. The user selects a virtual sales associate and begins interacting with them."

[0527] In this way, users can enjoy a more emotional and face-to-face interaction, making it possible to provide a high level of satisfaction that was difficult to achieve with conventional customer service systems.

[0528] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0529] Step 1:

[0530] The server captures the operator's voice. Specifically, when the operator speaks into the microphone, the voice is sent to the server as a digital signal. This voice data is analyzed using digital signal processing (DSP) and saved in a database in real time. The input is the operator's voice through the microphone, and the output is the analyzed digital voice data.

[0531] Step 2:

[0532] The server generates an animated character based on the operator's voice data. It uses a generative AI model (e.g., DALL-E or MidJourney) to generate an image of the animated character from the voice features. This generated character is stored in a database. The input is the operator's voice sample, and the output is an image of the generated animated character.

[0533] Step 3:

[0534] The server synchronizes the generated animated character with the operator's voice in real time. To do this, it uses real-time voice analysis software to analyze the voice data and match the character's mouth movements and facial expressions. The input is digital voice data and an image of the animated character, and the output is a synchronized set of animated character and voice.

[0535] Step 4:

[0536] The server transmits synchronized audio and animated characters to the user device. The communication protocol is HTTP or WebSocket, and data is streamed to the user device in real time. The input is synchronized animated character and audio data, and the output is real-time animation displayed on the user device.

[0537] Step 5:

[0538] The terminal sends a connection request from the user to the server. Specifically, when the user launches an application on their smartphone or smart glasses and presses the connect button, the request is sent to the server. The input is the user's connection request, and the output is the connection request data sent to the server.

[0539] Step 6:

[0540] The server sends a list of waiting operators to the user terminal. It retrieves operator information from the database, generates a list of available operators, and sends it to the user terminal. The input is the operator information stored in the database, and the output is the list of waiting operators displayed on the user terminal.

[0541] Step 7:

[0542] The user selects an operator and sends the selected information to the server through the terminal. The input is the user's selection operation, and the output is the selected information sent to the server.

[0543] Step 8:

[0544] The server analyzes the user's facial expression data received from the device to recognize the user's emotions. DeepFace is used for emotion recognition, and the analysis results are reflected in the character's facial expressions. The input is the user's facial expression data, and the output is the recognized emotional data and the character's facial expressions that change based on it.

[0545] Step 9:

[0546] The server records interactions and collects feedback. The content of the dialogue with the user is saved as digital data and later analyzed. This data is analyzed by the feedback collection system and used to improve the service. The input is the dialogue data with the user, and the output is the analysis results as feedback.

[0547] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0548] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0549] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0550] [Second embodiment]

[0551] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0552] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0553] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0554] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0555] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0556] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0557] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0558] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0559] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0560] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0561] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0562] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0563] The system according to the present invention synchronizes the voice of an operator with an animated character in real time, providing users with a more face-to-face interaction. This system can be implemented through the following steps:

[0564] server

[0565] 1. Register operator information

[0566] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[0567] The server sends a request to the image generation AI using the registered voice sample to generate an animated character of the operator.

[0568] The generated animated character is again saved in the database.

[0569] 2. Waiting for client connections

[0570] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[0571] The server maintains and updates a list of available operators.

[0572] 3. Operator selection and synchronization

[0573] When the server receives a request from a client, it sends a list of waiting operators to the client.

[0574] Once the client selects the desired operator, the information is returned to the server, which synchronizes the voice and animated character of the selected operator in real time.

[0575] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animated character.

[0576] 4. Sending voice and characters

[0577] The server transmits synchronized audio and animated characters to the client in real time.

[0578] Terminal (client)

[0579] 1. Sending a connection request

[0580] Users use their devices to send connection requests to a call center, which can be made by phone or through a dedicated application.

[0581] 2. Operator Selection

[0582] The list of waiting operators sent from the server is displayed on the terminal.

[0583] The user selects a desired animated character from the list and transmits the selection information to the server.

[0584] 3. Real-time synchronization

[0585] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[0586] 4. Recording interactions

[0587] After the interaction, the user's interaction is recorded and used to collect feedback and satisfaction.

[0588] Specific examples

[0589] Example of a system where Operator A is registered

[0590] Registering operator information

[0591] 1. The server registers Operator A's name, profile picture, and voice sample.

[0592] 2. The server uses operator A's voice sample to request the image generation AI to generate an animated character.

[0593] 3. The generated animated character is saved in the database.

[0594] User connects to call center

[0595] 1. A user connects to a call center using a phone or application.

[0596] 2. The server sends a list of waiting operators to the user terminal.

[0597] 3. The user selects an animated character for Operator A and sends that information back to the server.

[0598] Real-time synchronization begins

[0599] 1. The server captures operator A's voice in real time and synchronizes it with the movement data of the animation character that corresponds to that voice.

[0600] 2. Synchronized audio and animated characters are transmitted to the user's device in real time.

[0601] 3. Users can receive service as if they were face-to-face with Operator A through the animated character.

[0602] This system allows users to enjoy a more emotional and face-to-face service compared to traditional call center services. Furthermore, by recording interactions and collecting feedback, it is possible to improve the quality of service.

[0603] The processing flow will be explained below.

[0604] Server Processing Steps

[0605] Step 1:

[0606] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[0607] Step 2:

[0608] The server sends the voice samples to an image generation AI to generate an animated character for the operator.

[0609] Step 3:

[0610] The server stores the generated animated characters in a database.

[0611] Step 4:

[0612] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[0613] Step 5:

[0614] The server maintains an updated list of available operators.

[0615] Step 6:

[0616] The server receives a connection request from a client and sends a list of waiting operators to the client.

[0617] Step 7:

[0618] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[0619] Step 8:

[0620] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animation character.

[0621] Step 9:

[0622] The server transmits synchronized audio and animated characters to the client in real time.

[0623] Terminal (client) processing steps

[0624] Step 1:

[0625] The user uses a terminal to send a connection request to the call center, which can be made over the phone or through an application.

[0626] Step 2:

[0627] The list of waiting operators sent from the server is displayed on the terminal.

[0628] Step 3:

[0629] The user selects a desired animated character from the list and transmits the selection information to the server.

[0630] Step 4:

[0631] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[0632] Step 5:

[0633] After the interaction, the user's interaction is recorded and used to collect feedback and satisfaction.

[0634] User processing steps

[0635] Step 1:

[0636] The user connects to the call center by phone or using an application.

[0637] Step 2:

[0638] After connecting, check the list of waiting operators and select the character you want.

[0639] Step 3:

[0640] The selected animated character will synchronize with the audio in real time and begin interacting.

[0641] Step 4:

[0642] After the interaction, use the feedback form to provide your satisfaction and opinions.

[0643] Example 1

[0644] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0645] In conventional call center systems, users communicate with operators only through voice, which means there is a lack of visual information, making it difficult to convey emotions and facial expressions compared to face-to-face interactions. Furthermore, the limited options for operators can lead to lower user satisfaction. Furthermore, the lack of interaction recording and feedback collection functions poses a challenge, resulting in a lack of data to improve service quality.

[0646] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0647] In this invention, the server includes a means for acquiring the voice of an operator, a means for generating a prompt sentence for a generative AI model based on the voice and generating an animated character, a means for synchronizing the generated animated character with the voice of the operator in real time, a means for transmitting the synchronized voice and character to the client, and a means for the user to select an animated character for the operator. This allows users to enjoy a visually rich, face-to-face interaction, and the increased choice of operators increases satisfaction. Furthermore, by recording interactions and collecting feedback, the quality of service can be continuously improved.

[0648] "Operator" means a person who provides services and support to users via telecommunications.

[0649] "Voice" refers to data of the voice or speech produced by the operator.

[0650] A "generative AI model" is an artificial intelligence algorithm or program that generates new data based on input data. Specifically, it refers to a model for generating animated characters from voice or text.

[0651] A "prompt" is text that is input to a generative AI model and contains instructions and conditions for obtaining a specific output.

[0652] An "animated character" is a virtual figure or character generated based on the operator's voice and movements.

[0653] "Real-time" means a state in which delays are extremely small and immediate responses are possible.

[0654] "Synchronization" means moving or matching multiple pieces of data or actions simultaneously.

[0655] A "client" refers to a terminal or software for using the system, and is connected to the server primarily to process user requests.

[0656] A "database" is a collection of information organized to efficiently store, retrieve, and update data.

[0657] "Interaction" is the process of exchange or operation between a user and a system or operator.

[0658] "Feedback" refers to the evaluations and opinions provided by users regarding the quality and usability of the Service, and includes the process of improving the Service based on the collected information.

[0659] The system according to the present invention synchronizes the voice of an operator with an animated character in real time, providing users with a service that is almost like a face-to-face interaction. This system is configured as follows.

[0660] server

[0661] The server registers the operator's basic information (name, profile photo, voice sample) in a database. After registration, the voice sample is used to generate a prompt for a generative AI model (e.g., DALL-E or Stable Diffusion) to generate an animated character. Specifically, an example of a prompt might be, "Based on this voice sample, please generate a real-time animated character for Operator A." The generated animated character is saved in the database.

[0662] The server waits for a connection request from the user. Once a connection is made, it sends a list of available operators to the client. Once the user selects an operator, it uses this information to synchronize the voice and animated characters in real time and transmits them to the client using a low-latency protocol (e.g., WebRTC).

[0663] Terminal (client)

[0664] The user uses their device to send a connection request to the call center through a dedicated application or browser. Once connected, a list of available operators sent from the server is displayed, and the user selects the desired operator. The selection information is sent back to the server, and real-time voice and animated characters are displayed on the device. The necessary data is received via a low-latency protocol such as WebRTC and rendered in real time.

[0665] User

[0666] Users can easily connect to the call center using a dedicated application or browser and select an operator. After the call is completed, the interaction is recorded and users can provide feedback to improve the quality of the service. This interaction and feedback are stored in a database and used to improve the service.

[0667] Specific examples

[0668] An example will be explained in which operator A is newly registered.

[0669] 1. Register operator information

[0670] The server registers Operator A's name, profile picture and voice sample in a database.

[0671] The server uses the voice sample to generate a prompt for the generation AI (e.g., Stable Diffusion), specifically, the prompt sentence "Based on this voice sample, please generate a real-time animated character of Operator A."

[0672] The generated animated characters are stored in a database.

[0673] 2. The user connects to the call center

[0674] Users use a dedicated application to connect to the call center by sending an HTTP request.

[0675] The server sends a list of waiting operators to the user terminal.

[0676] The user selects the desired operator and returns the selection information to the server.

[0677] 3. Start syncing in real time

[0678] The server captures the audio data in real time and synchronizes it with the animated character using an AI animation engine (e.g., Live2D).

[0679] The server sends the synchronized data to the client using WebRTC.

[0680] The user can view the transmitted audio and animated characters in real time using a dedicated application.

[0681] In this way, the system of the present invention allows users to enjoy visually rich, face-to-face interaction, improving the quality of service.

[0682] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0683] Step 1: Register operator information

[0684] The server registers basic information of the operator (name, profile photo, voice sample) in the database. As input, it receives the operator's name, profile photo, and voice sample. In concrete terms, it collects the voice sample and stores it in the "operators" table of the database along with the name and photo. The output is the operator information registered in the database.

[0685] Step 2: Sending a request to the image generation AI

[0686] The server uses the registered voice sample to generate a prompt for a generative AI model (e.g., DALL-E or Stable Diffusion) and generates an animated character. The input in this step is the voice sample, and an example prompt is "Please generate a real-time animated character for Operator A based on this voice sample." Specifically, the server sends a prompt to the generative AI model to generate an animated character. The output is data for the generated animated character, which is stored in a database.

[0687] Step 3: Waiting for client connections

[0688] The server waits for connection requests from users. The input includes user connection requests, which may come over telephone lines or the Internet. Specific operations include listening on a TCP / IP port or using a telephony API. The output is a list of connected clients.

[0689] Step 4: Submit the Operator List

[0690] When the server receives a connection request, it sends a list of waiting operators to the client. The input in this step is the waiting operator information retrieved from the database, and this information is sent to the client in JSON format. The output is the operator list displayed on the user's terminal.

[0691] Step 5: Receive operator selection information

[0692] When the user selects the desired operator, the selection information is sent back to the server. The input of this step is the operator information selected by the user, specifically, the selection information is sent in JSON format. The output is the selection information received by the server.

[0693] Step 6: Synchronizing the audio with the animated character

[0694] The server captures the selected operator's voice in real time and synchronizes it with the corresponding animated character's movement data. The input in this step is real-time voice data and the generated animated character's movement data. Specifically, the voice data is synchronized using a streaming protocol (e.g., WebRTC), and the movement data is synchronized using an AI animation engine (e.g., Live2D). The output is the synchronized voice and animated character data.

[0695] Step 7: Sending voice and animated characters

[0696] The server sends synchronized audio and animated character data to the client in real time. The input in this step is the synchronized audio and animated character data, specifically, the transmission uses WebSocket or other low-latency protocols. The output is the data sent to the client.

[0697] Step 8: Sending a Connection Request

[0698] The user uses a terminal to send a connection request to the call center through a dedicated application or browser. The input to this step is the user's connection request, specifically, using an HTTP request or WebSocket. The output is the connection request sent to the server.

[0699] Step 9: Select an operator

[0700] The list of waiting operators sent from the server is displayed on the terminal. The user selects the desired operator from the list and sends the selection information to the server. The input in this step is the list of waiting operators, and the output is the operator information selected by the user.

[0701] Step 10: Viewing Real-Time Sync

[0702] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal. The input is synchronized data transmitted from the server, specifically received via a low-latency protocol (e.g., WebRTC). The output is the animated character and voice displayed on the user's terminal.

[0703] Step 11: Record the interaction

[0704] Once the call is over, the user interaction is recorded. The input to this step is the data of the user interaction, specifically stored in the "interactions" table in the database. The output is the recorded interaction data.

[0705] (Application example 1)

[0706] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0707] Conventional call center systems rely mainly on voice communication, and lack visual interaction with users, resulting in lower user satisfaction compared to face-to-face support. Furthermore, there are also challenges such as the difficulty of providing prompt and friendly responses in emergencies and security-related inquiries.

[0708] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0709] In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the voice of the operator in real time, means for transmitting the synchronized voice and character to a user terminal, means for a user to select an operator, and means for applying the character display and voice synchronization to emergency calls and inquiries. This allows users to communicate with operators in real time in a visually friendly manner. This also allows for a rapid response in security-related emergencies.

[0710] "Operator" refers to a staff member who responds to user inquiries.

[0711] "Means for capturing audio" refers to a device or software for capturing the operator's speech in real time.

[0712] "Animated Character" refers to a dynamic character that is generated based on the operator's voice and visually displayed to the user.

[0713] "Real-time synchronization means" means any device or software that synchronizes the voice of an operator with the movements of an animated character.

[0714] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0715] The term "means for a user to select an operator" refers to an interface or function that allows a user to select any operator from a plurality of operators.

[0716] "Applicable emergency communication and inquiry response measures" refers to the functions or processes that ensure efficient response to security-related emergencies and other inquiries.

[0717] The system for implementing this invention is composed of a server, a client terminal, and a user as its main components. The overall flow is that the server registers operator information, and based on that information, synchronizes the operator's voice with an animated character in real time and transmits it to the user terminal.

[0718] Server Configuration

[0719] The server consists of the following hardware and software:

[0720] Hardware: Server equipped with high-performance CPU, memory, storage, etc. (e.g. Dell PowerEdge)

[0721] Software: Python and Django for server-side processing, SQL for database management, TensorFlow and custom AI models for image generation and audio synchronization

[0722] function:

[0723] 1. Operator information registration:

[0724] The operator's name, profile picture and voice sample are entered into a database.

[0725] Using a voice sample, we request the image generation AI to generate an animated character.

[0726] The generated animated character is saved in a database.

[0727] 2. Waiting for client connections:

[0728] Waits for call center connection requests via telephone lines or the Internet.

[0729] 3. Operator selection and synchronization:

[0730] Receives requests from clients and sends a list of waiting operators to the user's terminal.

[0731] Real-time synchronization of voice and animated characters for selected operators.

[0732] 4. Voice and character transmission:

[0733] Synchronized voice and animated characters are transmitted to a user terminal in real time.

[0734] Client terminal

[0735] Client devices are primarily expected to be smartphones. Applications are developed and operated in the following environments:

[0736] Hardware: Android smartphone or iPhone

[0737] Software: Kotlin (for Android), Swift (for iOS), Unity (for animation display)

[0738] function:

[0739] 1. Send a connection request:

[0740] The user uses the application on the terminal to send a connection request to the call center.

[0741] 2. Operator Selection:

[0742] The list of waiting operators sent from the server is displayed on the terminal.

[0743] Provides an interface for users to select their preferred operator.

[0744] 3. Real-time Sync:

[0745] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[0746] 4. Recording interactions:

[0747] After the interaction, the user's interaction is recorded to collect feedback and satisfaction.

[0748] Specific examples

[0749] When a user detects an abnormality in their home security system during an emergency, they connect to the "Security Assistant" through the app. A list of operators is displayed, and the user can select one. The selected operator's animated character will appear in real time, and emergency response instructions will be given through synchronized voice.

[0750] Prompt Sentence Examples

[0751] Examples of prompts include:

[0752] "Please register your operator information. The operator's name is "Taro", the profile picture is the attached image, and the voice sample is the attached audio file. Please generate an animated character based on this information."

[0753] This configuration allows users to communicate visually with operators in real time and receive prompt and accurate responses, and also enables rapid response in security-related emergencies.

[0754] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0755] Step 1:

[0756] The server registers the operator's name, profile picture, and voice sample in the database. The input of this step is the operator's basic information, and the output is the operator's information stored in the database. Data processing involves analyzing the input voice sample, converting it into an appropriate format, and saving it.

[0757] Step 2:

[0758] The server uses the registered voice sample to request the image generation AI to generate an animated character. The input of this step is the voice sample, and the output is the generated animated character. Data calculation involves extracting voice features and inputting them into the image generation AI to generate the animated character.

[0759] Step 3:

[0760] The server saves the generated animated character in a database. The input of this step is the generated animated character, and the output is the character data saved in the database. Data processing involves converting the character data into an appropriate format and saving it.

[0761] Step 4:

[0762] The user uses the terminal to send a connection request to the call center. The input of this step is the user's connection request, and the output is the connection request information sent to the server. The specific operation is that the user opens the application and presses the call center connection button.

[0763] Step 5:

[0764] The server receives a call center connection request and sends a list of waiting operators to the user's terminal. The input of this step is the user's connection request, and the output is the list of waiting operators. Data processing involves converting the waiting operator information into a list format and sending it.

[0765] Step 6:

[0766] The user selects the desired operator from the list and sends the selection information to the server. The input of this step is the operator selection information, and the output is the selection information sent to the server. The specific operation is that the user taps on the terminal screen to select the operator.

[0767] Step 7:

[0768] The server synchronizes the voice of the selected operator with the animation character in real time. The input of this step is the operator's voice, and the output is the synchronized animation character. The data calculation is to analyze the voice data in real time and generate the animation movement accordingly.

[0769] Step 8:

[0770] The server transmits the synchronized voice and animated characters to the user terminal in real time. The input of this step is the synchronized voice and characters, and the output is the data transmitted to the user terminal. The data processing is to perform real-time data streaming.

[0771] Step 9:

[0772] The user interacts with the operator through an animated character displayed in real time on the terminal. The input of this step is the voice and character sent to the terminal, and the output is the user's interaction experience. The specific operation is for the user to continue the conversation with the operator while looking at the terminal screen.

[0773] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0774] The present invention relates to a system that combines real-time synchronization of an operator's voice with an animated character, and an emotion engine for recognizing the user's emotions. The system is implemented through the following steps:

[0775] server

[0776] 1. Register operator information

[0777] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[0778] The server sends a request to the image generation AI using the registered voice sample to generate an animated character of the operator.

[0779] The generated animated characters are stored in a database.

[0780] 2. Waiting for client connections

[0781] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[0782] The server maintains and updates a list of available operators.

[0783] 3. Operator selection and synchronization

[0784] The server receives a request from a client and sends a list of waiting operators to the client.

[0785] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[0786] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animated character.

[0787] 4. Emotion Recognition by Emotion Engine

[0788] The server uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[0789] The recognized emotion data is used to change the facial expressions and movements of the operator's animated character.

[0790] 5. Voice and character transmission

[0791] The server transmits synchronized audio and animated characters to the client in real time.

[0792] 6. Recording interaction and emotion data

[0793] The server records the user's interactions and the perceived emotion data and integrates it into subsequent feedback collection measures.

[0794] Terminal (client)

[0795] 1. Sending a connection request

[0796] Users use their devices to send connection requests to a call center, which can be made via telephone or a dedicated application.

[0797] 2. Operator Selection

[0798] The list of waiting operators sent from the server is displayed on the terminal.

[0799] The user selects a desired animated character from the list and transmits the selection information to the server.

[0800] 3. Real-time synchronization

[0801] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[0802] 4. Emotion-aware feedback

[0803] The terminal receives the user's emotion recognized by the emotion engine and provides an operator's response based on the emotion.

[0804] 5. Recording interactions

[0805] After the interaction, the user's interaction and emotional data is recorded and used to collect feedback and satisfaction.

[0806] User processing steps

[0807] 1. Connect to the call center

[0808] The user connects to the call center by phone or using an application.

[0809] 2. Confirm your operator selection

[0810] After connecting, check the list of waiting operators and select the character you want.

[0811] 3. Start of the conversation

[0812] The selected animated character will synchronize with the audio in real time and begin interacting.

[0813] 4. Providing emotional feedback

[0814] After the call is over, the emotional data recognized by the emotion engine is integrated into a feedback form to provide satisfaction and opinions.

[0815] Specific examples

[0816] Example of a system where Operator A is registered

[0817] Registering operator information

[0818] 1. The server registers Operator A's name, profile picture, and voice sample.

[0819] 2. The server uses operator A's voice sample to request the image generation AI to generate an animated character.

[0820] 3. The generated animated character is saved in the database.

[0821] User connects to call center

[0822] 1. A user connects to a call center using a phone or application.

[0823] 2. The server sends a list of waiting operators to the user terminal.

[0824] 3. The user selects an animated character for Operator A and sends that information back to the server.

[0825] Real-time synchronization begins

[0826] 1. The server captures operator A's voice in real time and synchronizes it with the corresponding animation character's movement data.

[0827] 2. Synchronized audio and animated characters are transmitted to the user's device in real time.

[0828] 3. The emotion engine recognizes the user's emotions and uses that data to change the facial expression of the operator's animated character.

[0829] 4. Users can receive service through the animated character as if they were face-to-face with Operator A.

[0830] This system allows users to enjoy a more emotional and face-to-face service compared to traditional call center services. Furthermore, by recording interactions and collecting feedback, it is possible to improve the quality of service.

[0831] The processing flow will be explained below.

[0832] Server Processing Steps

[0833] Step 1:

[0834] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[0835] Step 2:

[0836] The server sends the registered voice samples to an image generation AI to generate an animated character of the operator.

[0837] Step 3:

[0838] The server stores the generated animated characters in a database.

[0839] Step 4:

[0840] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[0841] Step 5:

[0842] The server maintains an updated list of available operators.

[0843] Step 6:

[0844] The server receives a connection request from a client and sends a list of waiting operators to the client.

[0845] Step 7:

[0846] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[0847] Step 8:

[0848] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animation character.

[0849] Step 9:

[0850] The server uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[0851] Step 10:

[0852] The server changes the facial expressions and movements of the operator's animated character based on the recognized emotional data.

[0853] Step 11:

[0854] The server transmits synchronized audio and animated characters to the client in real time.

[0855] Step 12:

[0856] The server records the user's interactions and the perceived emotion data and integrates it into subsequent feedback collection measures.

[0857] Terminal (client) processing steps

[0858] Step 1:

[0859] Users use their devices to send connection requests to a call center, which can be made via telephone or a dedicated application.

[0860] Step 2:

[0861] The list of waiting operators sent from the server is displayed on the terminal.

[0862] Step 3:

[0863] The user selects a desired animated character from the list and transmits the selection information to the server.

[0864] Step 4:

[0865] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[0866] Step 5:

[0867] The terminal receives the user's emotions recognized by the emotion engine and changes the facial expression of the operator's animated character in real time based on that information.

[0868] Step 6:

[0869] After answering the call, the terminal records the user's interaction and the recognized emotion data and transmits them to the server.

[0870] User processing steps

[0871] Step 1:

[0872] The user connects to the call center by phone or using an application.

[0873] Step 2:

[0874] After connecting, check the list of waiting operators and select the character you want.

[0875] Step 3:

[0876] The selected animated character will synchronize with the audio in real time and begin interacting.

[0877] Step 4:

[0878] Interact with an operator whose facial expressions change in real time thanks to an emotion engine.

[0879] Step 5:

[0880] After the call is over, the emotional data recognized by the emotion engine is integrated into a feedback form to provide satisfaction and opinions.

[0881] Through the above steps, the system can provide users with an emotional and face-to-face interaction, and the recording of interaction and emotional data provides valuable data for improving service quality.

[0882] Example 2

[0883] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] Conventional call center systems mainly rely on voice communication between operators and users, limiting interaction. Furthermore, they are unable to recognize users' emotions in real time and respond accordingly, resulting in a poor user experience. Furthermore, processes such as operator selection and interaction recording are inefficient, making it difficult to improve service quality. There is a need to solve these problems and provide responses that are more emotional and closer to face-to-face interactions.

[0885] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0886] In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the operator's voice in real time, means for transmitting the synchronized voice and character to a user terminal, means for the user to select an operator, means for recognizing the user's emotion, and means for changing the facial expressions and movements of the animated character based on the recognized emotion. This allows the user to receive emotional responses in real time, allowing for an interaction experience that is closer to face-to-face interaction. Furthermore, feedback can be collected based on interaction records and emotion data, making it possible to continuously improve the quality of service.

[0887] "Means for acquiring the operator's voice" refers to devices or software for capturing the voice data spoken by the operator in real time and inputting it into the system.

[0888] "Means for generating animated characters" means technologies and algorithms for generating animated characters that can move in real time based on operator voice and profile data.

[0889] "Means for synchronizing voice and characters in real time" refers to software or systems that allow for timely adjustment and reflection of the movements and expressions of animated characters based on the operator's voice data.

[0890] "Means for transmitting to the user terminal" refers to the communication technology and protocol for transmitting synchronized audio data and animation data from the server to the client terminal in real time.

[0891] "Means for the user to select an operator" refers to a GUI (graphical user interface) on the interface used by the user to select a desired operator from a plurality of operators.

[0892] "Means for recognizing user emotions" refers to an emotion recognition engine or algorithm that processes the user's voice data and facial expression data and analyzes their emotional state.

[0893] "Means for changing the facial expressions and movements of an animated character based on emotions" refers to a control system or algorithm that uses recognized user emotional data to change the facial expressions and movements of an animated character in real time.

[0894] "Means for storing operator information in a database" refers to the functions and systems for registering and storing basic information about operators (such as names, profile photos, and voice samples) in a database.

[0895] "Means for sending a list of waiting operators to a client" refers to a communication means or protocol for generating a list of waiting operators on the server side and sending it to the user's terminal.

[0896] "Means for recording interactions" refers to technologies and systems that record all voice interactions and actions between users and operators for later analysis and evaluation.

[0897] "Means for collecting feedback based on emotional data" refers to a method for analyzing emotional data collected during interactions with users and using the results to collect user satisfaction and areas for improvement through a feedback form, etc.

[0898] "Measures to improve service quality" refers to continuous improvement methods that analyze collected feedback data and use it to improve the way operators respond and the performance of the entire system.

[0899] MODE FOR CARRYING OUT THE INVENTION

[0900] This invention relates to a system that synchronizes voice and animation in real time and recognizes a user's emotions. This system is composed of a server, a terminal (client), and a user.

[0901] Basic Server Configuration

[0902] The server is the central component responsible for the main processing and implements the following functions:

[0903] 1. Register operator information

[0904] The server stores various information about the operator (name, profile photo, voice sample) in a database. An RDBMS such as MySQL is used here. Using Python's OpenAI API library, the voice sample is sent to a generative AI model (e.g., DALL-E) to generate an animated character. The generated character is then stored back in the database.

[0905] 2. Waiting for client connections

[0906] The server waits for and processes connection requests from clients using a communications platform such as Twilio.

[0907] 3. Operator selection and synchronization

[0908] Based on the client's request, the server sends a list of available operators to the client. When the user selects an operator, the server receives that information and synchronizes the operator's voice data with the animated character in real time. APIs such as Amazon Polly and Google Text-to-Speech are used for voice processing.

[0909] 4. Emotion Recognition by Emotion Engine

[0910] The server uses Azure Cognitive Services' Emotion API to recognize emotions from the user's voice and facial expressions, and the recognized emotional data is reflected in the facial expressions and movements of the animated character in real time.

[0911] 5. Voice and character transmission

[0912] The server sends synchronized audio and animated characters to the client using real-time communication such as WebSocket.

[0913] 6. Recording interaction and emotion data

[0914] The server records user interactions and emotional data and stores it in Amazon S3 or Google Cloud Storage, which helps gather subsequent feedback and improve the service.

[0915] Basic configuration of the terminal (client)

[0916] The client terminal is a device used by a user and has the following functions:

[0917] 1. Sending a connection request

[0918] Users use a dedicated smartphone app or web application to send a connection request to a server, which is established through a service such as Twilio.

[0919] 2. Operator Selection

[0920] The user checks the list of waiting operators sent from the server and selects the desired operator, and the selection information is sent to the server.

[0921] 3. Real-time synchronization

[0922] The client receives and displays the selected operator's voice and animated character in real time. The animated character moves in real time using the Unity engine or similar.

[0923] 4. Emotion-aware feedback

[0924] The client receives the user's emotion data recognized by the emotion engine and displays a response based on that data (such as changing the character's facial expression).

[0925] 5. Recording interactions

[0926] The client records the user's interaction and emotion data and sends it to the server, which uses the data to improve satisfaction and services.

[0927] Basic user actions

[0928] A user uses the system in the following steps:

[0929] 1. Connect to the call center

[0930] Users connect to the call center using a phone or an application, with the connection provided through a communications platform such as Twilio.

[0931] 2. Confirm your operator selection

[0932] The user checks the list of waiting operators presented by the server and selects a desired operator.

[0933] 3. Start of the conversation

[0934] The selected animated character will be synchronized with the voice in real time, allowing the user to receive a face-to-face response, and the emotion engine will recognize the user's emotions in real time, providing a more emotionally relevant response.

[0935] 4. Providing emotional feedback

[0936] After the call is completed, a feedback form is displayed based on the emotional data recognized by the emotion engine, allowing the user to provide their satisfaction and opinions.

[0937] Specific examples

[0938] Prompt Sentence Examples

[0939] Example: Use the voice sample of Operator A to generate an animated character using a generative AI model. Voice sample URL: http: / / example.com / audio_sample

[0940] Compared to traditional call centers, the system offers a more empathetic, face-to-face experience, and by recording interactions and collecting feedback, it can continuously improve service quality.

[0941] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0942] Program processing flow

[0943] server

[0944] Step 1: Register operator information

[0945] Input: Operator name, profile picture, voice sample

[0946] What it does: The server receives the operator's name, profile picture, and voice sample from the form and stores the data in a MySQL database.

[0947] Data processing: The server uses Python's OpenAI API library to send voice samples to DALL-E with prompt sentences to generate a corresponding animated character.

[0948] Output: The generated animated characters are saved back to the database.

[0949] Step 2: Waiting for client connections

[0950] Input: Connection request from the client

[0951] Specific operation: The server waits for a call center connection request from a client using a cloud communication platform such as Twilio.

[0952] Data Processing: Manage and update the list of available operators in real time.

[0953] Output: Generates the operator list upon receiving a request from the client.

[0954] Step 3: Select and sync operators

[0955] Input: Operator selection information from the user

[0956] Specific operation: Based on a request from a client, the server sends a list of waiting operators to the client.

[0957] Data processing: The voice of the selected operator is synchronized with the movement data of the corresponding animated character in real time. Amazon Polly is used for voice processing.

[0958] Output: Synchronized data is generated.

[0959] Step 4: Emotion Recognition with the Emotion Engine

[0960] Input: User's voice and facial expressions

[0961] Specific operation: The server uses the Emotion API of Azure Cognitive Services to recognize emotions from the user's voice and facial expressions.

[0962] Data processing: Collect recognized emotion data and update the facial expressions and movements of the animated character.

[0963] Output: Updated animated character data is generated.

[0964] Step 5: Submit your voice and character

[0965] Input: Synchronized audio and animated character data

[0966] Specific operation: The server transmits synchronized audio and animated characters to the client in real time using WebSocket.

[0967] Data processing: Convert the data into a WebSocket communication format.

[0968] Output: Generates data sent to the client.

[0969] Step 6: Recording interaction and emotion data

[0970] Input: User interaction data and recognized emotion data

[0971] Specific Operation: The server records the user's interactions and the recognized emotion data.

[0972] Data processing: The recorded data is stored on Amazon S3 or Google Cloud Storage.

[0973] Output: The saved data is generated.

[0974] Terminal (client)

[0975] Step 1: Sending a Connection Request

[0976] Input: A user requesting a connection

[0977] How it works: A user sends a connection request to a server using a dedicated smartphone app or web application. This connection is made through a service such as Twilio.

[0978] Data processing: Generate connection request data.

[0979] Output: Generates the connection request data sent to the server.

[0980] Step 2: Select an operator

[0981] Input: List of waiting operators from the server

[0982] Specific operation: The user checks the list of waiting operators sent from the server and selects the desired operator. The selection information is sent to the server.

[0983] Data processing: Generate user selection information.

[0984] Output: Selection information is generated that is sent to the server.

[0985] Step 3: Real-time sync

[0986] Input: Synchronized audio and animated character data sent from the server

[0987] Specific operation: The client receives and displays the selected operator's voice and animated character in real time. The animated character moves in real time using the Unity engine or similar.

[0988] Data processing: Converting received data into a playable format.

[0989] Output: Synchronized voice and character data is generated that is displayed to the user.

[0990] Step 4: Emotion recognition feedback

[0991] Input: Emotion data from the server

[0992] Specific operation: The client receives the user emotion data recognized by the emotion recognition engine and displays a response based on that data (such as changing the character's facial expression).

[0993] Data processing: Generate animation update data based on emotion data.

[0994] Output: Updated animated character data is generated that is displayed to the user.

[0995] Step 5: Record the interaction

[0996] Input: User interaction data and emotion data

[0997] Specific operation: After the interaction, the client records the user's interaction data and emotion data and sends them to the server.

[0998] Data processing: Generate recorded interaction data and emotion data.

[0999] Output: Interaction data and emotion data are generated and sent to the server.

[1000] User

[1001] Step 1: Connect to a call center

[1002] Input: A user requesting a connection

[1003] What it does: A user connects to a call center using a phone or a dedicated app, and the connection is made through a communications platform such as Twilio.

[1004] Data processing: Generate connection request data and send it to the server.

[1005] Output: The connection request data sent to the server is generated.

[1006] Step 2: Confirm operator selection

[1007] Input: List of waiting operators from the server

[1008] Specific operation: The user checks the list of waiting operators presented by the server and selects the desired operator.

[1009] Data processing: Generate user selection information and send it to the server.

[1010] Output: Selection information is generated that is sent to the server.

[1011] Step 3: Start the conversation

[1012] Input: Synchronized audio and animated character data sent from the server

[1013] Specific operation: An animated character selected by the user is synchronized with the audio in real time, and the user is treated as if they were face-to-face.

[1014] Data processing: Converting received audio and animation data into a playable format.

[1015] Output: Synchronized voice and character data is generated that is displayed to the user.

[1016] Step 4: Provide emotional feedback

[1017] Input: Emotion data from the server

[1018] Specific operation: Based on the emotion data recognized by the emotion engine, a feedback form is displayed on the user's device. The user provides their satisfaction and opinions.

[1019] Data processing: Generate user feedback data and send it to the server.

[1020] Output: Feedback data is generated that is sent to the server.

[1021] (Application example 2)

[1022] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1023] Conventional customer service systems have the problem that the operator's facial expressions and emotions are difficult to convey, making communication with the user feel mechanical and cold. Furthermore, the lack of a system that can flexibly respond to the user's choices and emotions makes it difficult to provide highly satisfying service. This is particularly true in brick-and-mortar stores, where it is difficult for operators in remote locations to provide effective customer service in real time.

[1024] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the voice of the operator in real time, means for transmitting the synchronized voice and character to a user terminal, means for the user to select an operator, means for recognizing the user's emotion, and means for changing the facial expression of the animated character based on the recognized emotion. This allows the user to enjoy an emotional response similar to that of a face-to-face interaction, thereby enabling highly satisfying customer service.

[1025] "Means for acquiring the operator's voice" means a device or software for receiving the voice signal emitted by the operator and recording and processing it as digital data.

[1026] "Means for generating animated characters" means algorithms or software for automatically creating 2D or 3D animated characters based on operator voice or other data.

[1027] "Means of real-time synchronization" refers to technology or systems that instantly match the operator's voice and the movements of the animated character, displaying them without any time lag.

[1028] The "means for transmitting to the user terminal" is a communication function for transmitting the generated animated character and the audio synchronized therewith to the user's device via a network.

[1029] "Means for a user to select an operator" refers to an interface or system that allows a user to select a specific operator from among multiple operators.

[1030] "Means for recognizing user emotions" refers to algorithms and implementations for analyzing and identifying the emotional state of a user from their facial expressions and voice.

[1031] "Means for changing the facial expression of an animated character" refers to a function or system for appropriately changing the facial expression or movement of an animated character in accordance with the recognized emotions of the user.

[1032] "Means for storing operator information in a database" refers to a database system that centrally manages and stores basic information about operators, voice samples, generated character information, etc.

[1033] The "means for transmitting a list of waiting operators to a client" refers to a function or technology for transmitting a list of currently available operators to a user's device.

[1034] The "means for generating data for simulating information about a product" is a simulation function for automatically generating data required for product explanations and demonstrations.

[1035] The "means for providing product information to users" refers to a display function or interface for effectively conveying the generated product data to users.

[1036] "Means for recording user interactions" refers to a system for saving the content of conversations and actions with users as digital data.

[1037] "Means of collecting feedback" are surveys and rating systems for collecting user satisfaction and opinions.

[1038] "Means for dynamically changing customer service content based on user interaction" refers to algorithms and technologies that adapt and change customer service content in real time according to the user's dialogue and behavior.

[1039] The present invention relates to a system for recognizing a user's emotions by synchronizing an operator's voice with an animated character in real time. A specific embodiment of this system will now be described.

[1040] server

[1041] The server first captures the operator's voice. Commonly used hardware includes a microphone to record the voice as digital data and a server computer to process it. The voice data is captured using digital signal processing (DSP) technology.

[1042] The server then generates an animated character based on the voice data. The software used is a generative AI model (e.g., OpenAI's DALL-E or MidJourney). It analyzes the voice sample and generates an animated character based on its characteristics. This generated character is then stored in a database.

[1043] The server analyzes the voice data and links it with the character's movement data to synchronize the generated animated character and the operator's voice in real time. This synchronization is achieved using real-time voice analysis software and motion capture technology.

[1044] The server has a communication function to transmit synchronized audio and animated characters to the user terminal, using the commonly used Internet Protocol (TCP / IP) as the communication protocol.

[1045] The server also recognizes the user's emotions and dynamically changes the facial expression of the animated character based on that data. This emotion recognition uses the Hugging Face transformers library and DeepFace. The server analyzes emotions from the user's voice and facial expressions, dynamically changing the character's facial expression accordingly.

[1046] In addition, the server stores operator information in a database and has the function of sending a list of waiting operators to the user's terminal. It also plays a role in improving the efficiency of customer service by generating data to simulate product information and providing it to the user.

[1047] Terminal (client)

[1048] The user terminals include smartphones and smart glasses, and connect to the call center using these terminals. The user can send a connection request through the application and check the list of waiting operators. This list is provided by the server, and the user can select the operator they want.

[1049] Once the selection is complete, the voice and animated character are synchronized from the server and displayed in real time on the user's device. The device is equipped with a camera and microphone to recognize the user's emotions. This emotional data is sent to the server, and the character's facial expression changes dynamically based on the content of the conversation.

[1050] User

[1051] Users interact with the virtual store clerk through the terminal, and the system also has the ability to record user interactions and collect feedback, which is used to improve subsequent services.

[1052] Specifically, when a user visits a store and launches the app on their smartphone, a list of available virtual clerks is displayed. The user selects a virtual clerk from the list and begins a conversation. The system then recognizes the user's emotions and changes the virtual clerk's response accordingly. The following is an example of a prompt sentence:

[1053] Example prompt sentence:

[1054] "The user launches the app and gets a list of available virtual sales associates. The user selects a virtual sales associate and begins interacting with them."

[1055] In this way, users can enjoy a more emotional and face-to-face interaction, making it possible to provide a high level of satisfaction that was difficult to achieve with conventional customer service systems.

[1056] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1057] Step 1:

[1058] The server captures the operator's voice. Specifically, when the operator speaks into the microphone, the voice is sent to the server as a digital signal. This voice data is analyzed using digital signal processing (DSP) and saved in a database in real time. The input is the operator's voice through the microphone, and the output is the analyzed digital voice data.

[1059] Step 2:

[1060] The server generates an animated character based on the operator's voice data. It uses a generative AI model (e.g., DALL-E or MidJourney) to generate an image of the animated character from the voice features. This generated character is stored in a database. The input is the operator's voice sample, and the output is an image of the generated animated character.

[1061] Step 3:

[1062] The server synchronizes the generated animated character with the operator's voice in real time. To do this, it uses real-time voice analysis software to analyze the voice data and match the character's mouth movements and facial expressions. The input is digital voice data and an image of the animated character, and the output is a synchronized set of animated character and voice.

[1063] Step 4:

[1064] The server transmits synchronized audio and animated characters to the user device. The communication protocol is HTTP or WebSocket, and data is streamed to the user device in real time. The input is synchronized animated character and audio data, and the output is real-time animation displayed on the user device.

[1065] Step 5:

[1066] The terminal sends a connection request from the user to the server. Specifically, when the user launches an application on their smartphone or smart glasses and presses the connect button, the request is sent to the server. The input is the user's connection request, and the output is the connection request data sent to the server.

[1067] Step 6:

[1068] The server sends a list of waiting operators to the user terminal. It retrieves operator information from the database, generates a list of available operators, and sends it to the user terminal. The input is the operator information stored in the database, and the output is the list of waiting operators displayed on the user terminal.

[1069] Step 7:

[1070] The user selects an operator and sends the selected information to the server through the terminal. The input is the user's selection operation, and the output is the selected information sent to the server.

[1071] Step 8:

[1072] The server analyzes the user's facial expression data received from the device to recognize the user's emotions. DeepFace is used for emotion recognition, and the analysis results are reflected in the character's facial expressions. The input is the user's facial expression data, and the output is the recognized emotional data and the character's facial expressions that change based on it.

[1073] Step 9:

[1074] The server records interactions and collects feedback. The content of the dialogue with the user is saved as digital data and later analyzed. This data is analyzed by the feedback collection system and used to improve the service. The input is the dialogue data with the user, and the output is the analysis results as feedback.

[1075] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1076] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1077] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1078] [Third embodiment]

[1079] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1080] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1081] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1082] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1083] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1084] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1085] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1086] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1087] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1088] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1089] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1090] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1091] The system according to the present invention synchronizes the voice of an operator with an animated character in real time, providing users with a more face-to-face interaction. This system can be implemented through the following steps:

[1092] server

[1093] 1. Register operator information

[1094] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[1095] The server sends a request to the image generation AI using the registered voice sample to generate an animated character of the operator.

[1096] The generated animated character is again saved in the database.

[1097] 2. Waiting for client connections

[1098] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[1099] The server maintains and updates a list of available operators.

[1100] 3. Operator selection and synchronization

[1101] When the server receives a request from a client, it sends a list of waiting operators to the client.

[1102] Once the client selects the desired operator, the information is returned to the server, which synchronizes the voice and animated character of the selected operator in real time.

[1103] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animated character.

[1104] 4. Sending voice and characters

[1105] The server transmits synchronized audio and animated characters to the client in real time.

[1106] Terminal (client)

[1107] 1. Sending a connection request

[1108] Users use their devices to send connection requests to a call center, which can be made by phone or through a dedicated application.

[1109] 2. Operator Selection

[1110] The list of waiting operators sent from the server is displayed on the terminal.

[1111] The user selects a desired animated character from the list and transmits the selection information to the server.

[1112] 3. Real-time synchronization

[1113] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[1114] 4. Recording interactions

[1115] After the interaction, the user's interaction is recorded and used to collect feedback and satisfaction.

[1116] Specific examples

[1117] Example of a system where Operator A is registered

[1118] Registering operator information

[1119] 1. The server registers Operator A's name, profile picture, and voice sample.

[1120] 2. The server uses operator A's voice sample to request the image generation AI to generate an animated character.

[1121] 3. The generated animated character is saved in the database.

[1122] User connects to call center

[1123] 1. A user connects to a call center using a phone or application.

[1124] 2. The server sends a list of waiting operators to the user terminal.

[1125] 3. The user selects an animated character for Operator A and sends that information back to the server.

[1126] Real-time synchronization begins

[1127] 1. The server captures operator A's voice in real time and synchronizes it with the movement data of the animation character that corresponds to that voice.

[1128] 2. Synchronized audio and animated characters are transmitted to the user's device in real time.

[1129] 3. Users can receive service as if they were face-to-face with Operator A through the animated character.

[1130] This system allows users to enjoy a more emotional and face-to-face service compared to traditional call center services. Furthermore, by recording interactions and collecting feedback, it is possible to improve the quality of service.

[1131] The processing flow will be explained below.

[1132] Server Processing Steps

[1133] Step 1:

[1134] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[1135] Step 2:

[1136] The server sends the voice samples to an image generation AI to generate an animated character for the operator.

[1137] Step 3:

[1138] The server stores the generated animated characters in a database.

[1139] Step 4:

[1140] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[1141] Step 5:

[1142] The server maintains an updated list of available operators.

[1143] Step 6:

[1144] The server receives a connection request from a client and sends a list of waiting operators to the client.

[1145] Step 7:

[1146] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[1147] Step 8:

[1148] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animation character.

[1149] Step 9:

[1150] The server transmits synchronized audio and animated characters to the client in real time.

[1151] Terminal (client) processing steps

[1152] Step 1:

[1153] The user uses a terminal to send a connection request to the call center, which can be made over the phone or through an application.

[1154] Step 2:

[1155] The list of waiting operators sent from the server is displayed on the terminal.

[1156] Step 3:

[1157] The user selects a desired animated character from the list and transmits the selection information to the server.

[1158] Step 4:

[1159] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[1160] Step 5:

[1161] After the interaction, the user's interaction is recorded and used to collect feedback and satisfaction.

[1162] User processing steps

[1163] Step 1:

[1164] The user connects to the call center by phone or using an application.

[1165] Step 2:

[1166] After connecting, check the list of waiting operators and select the character you want.

[1167] Step 3:

[1168] The selected animated character will synchronize with the audio in real time and begin interacting.

[1169] Step 4:

[1170] After the interaction, use the feedback form to provide your satisfaction and opinions.

[1171] Example 1

[1172] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1173] In conventional call center systems, users communicate with operators only through voice, which means there is a lack of visual information, making it difficult to convey emotions and facial expressions compared to face-to-face interactions. Furthermore, the limited options for operators can lead to lower user satisfaction. Furthermore, the lack of interaction recording and feedback collection functions poses a challenge, resulting in a lack of data to improve service quality.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1175] In this invention, the server includes a means for acquiring the voice of an operator, a means for generating a prompt sentence for a generative AI model based on the voice and generating an animated character, a means for synchronizing the generated animated character with the voice of the operator in real time, a means for transmitting the synchronized voice and character to the client, and a means for the user to select an animated character for the operator. This allows users to enjoy a visually rich, face-to-face interaction, and the increased choice of operators increases satisfaction. Furthermore, by recording interactions and collecting feedback, the quality of service can be continuously improved.

[1176] "Operator" means a person who provides services and support to users via telecommunications.

[1177] "Voice" refers to data of the voice or speech produced by the operator.

[1178] A "generative AI model" is an artificial intelligence algorithm or program that generates new data based on input data. Specifically, it refers to a model for generating animated characters from voice or text.

[1179] A "prompt" is text that is input to a generative AI model and contains instructions and conditions for obtaining a specific output.

[1180] An "animated character" is a virtual figure or character generated based on the operator's voice and movements.

[1181] "Real-time" means a state in which delays are extremely small and immediate responses are possible.

[1182] "Synchronization" means moving or matching multiple pieces of data or actions simultaneously.

[1183] A "client" refers to a terminal or software for using the system, and is connected to the server primarily to process user requests.

[1184] A "database" is a collection of information organized to efficiently store, retrieve, and update data.

[1185] "Interaction" is the process of exchange or operation between a user and a system or operator.

[1186] "Feedback" refers to the evaluations and opinions provided by users regarding the quality and usability of the Service, and includes the process of improving the Service based on the collected information.

[1187] The system according to the present invention synchronizes the voice of an operator with an animated character in real time, providing users with a service that is almost like a face-to-face interaction. This system is configured as follows.

[1188] server

[1189] The server registers the operator's basic information (name, profile photo, voice sample) in a database. After registration, the voice sample is used to generate a prompt for a generative AI model (e.g., DALL-E or Stable Diffusion) to generate an animated character. Specifically, an example of a prompt might be, "Based on this voice sample, please generate a real-time animated character for Operator A." The generated animated character is saved in the database.

[1190] The server waits for a connection request from the user. Once a connection is made, it sends a list of available operators to the client. Once the user selects an operator, it uses this information to synchronize the voice and animated characters in real time and transmits them to the client using a low-latency protocol (e.g., WebRTC).

[1191] Terminal (client)

[1192] The user uses their device to send a connection request to the call center through a dedicated application or browser. Once connected, a list of available operators sent from the server is displayed, and the user selects the desired operator. The selection information is sent back to the server, and real-time voice and animated characters are displayed on the device. The necessary data is received via a low-latency protocol such as WebRTC and rendered in real time.

[1193] User

[1194] Users can easily connect to the call center using a dedicated application or browser and select an operator. After the call is completed, the interaction is recorded and users can provide feedback to improve the quality of the service. This interaction and feedback are stored in a database and used to improve the service.

[1195] Specific examples

[1196] An example will be explained in which operator A is newly registered.

[1197] 1. Register operator information

[1198] The server registers Operator A's name, profile picture and voice sample in a database.

[1199] The server uses the voice sample to generate a prompt for the generation AI (e.g., Stable Diffusion), specifically, the prompt sentence "Based on this voice sample, please generate a real-time animated character of Operator A."

[1200] The generated animated characters are stored in a database.

[1201] 2. The user connects to the call center

[1202] Users use a dedicated application to connect to the call center by sending an HTTP request.

[1203] The server sends a list of waiting operators to the user terminal.

[1204] The user selects the desired operator and returns the selection information to the server.

[1205] 3. Start syncing in real time

[1206] The server captures the audio data in real time and synchronizes it with the animated character using an AI animation engine (e.g., Live2D).

[1207] The server sends the synchronized data to the client using WebRTC.

[1208] The user can view the transmitted audio and animated characters in real time using a dedicated application.

[1209] In this way, the system of the present invention allows users to enjoy visually rich, face-to-face interaction, improving the quality of service.

[1210] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1211] Step 1: Register operator information

[1212] The server registers basic information of the operator (name, profile photo, voice sample) in the database. As input, it receives the operator's name, profile photo, and voice sample. In concrete terms, it collects the voice sample and stores it in the "operators" table of the database along with the name and photo. The output is the operator information registered in the database.

[1213] Step 2: Sending a request to the image generation AI

[1214] The server uses the registered voice sample to generate a prompt for a generative AI model (e.g., DALL-E or Stable Diffusion) and generates an animated character. The input in this step is the voice sample, and an example prompt is "Please generate a real-time animated character for Operator A based on this voice sample." Specifically, the server sends a prompt to the generative AI model to generate an animated character. The output is data for the generated animated character, which is stored in a database.

[1215] Step 3: Waiting for client connections

[1216] The server waits for connection requests from users. The input includes user connection requests, which may come over telephone lines or the Internet. Specific operations include listening on a TCP / IP port or using a telephony API. The output is a list of connected clients.

[1217] Step 4: Submit the Operator List

[1218] When the server receives a connection request, it sends a list of waiting operators to the client. The input in this step is the waiting operator information retrieved from the database, and this information is sent to the client in JSON format. The output is the operator list displayed on the user's terminal.

[1219] Step 5: Receive operator selection information

[1220] When the user selects the desired operator, the selection information is sent back to the server. The input of this step is the operator information selected by the user, specifically, the selection information is sent in JSON format. The output is the selection information received by the server.

[1221] Step 6: Synchronizing the audio with the animated character

[1222] The server captures the selected operator's voice in real time and synchronizes it with the corresponding animated character's movement data. The input in this step is real-time voice data and the generated animated character's movement data. Specifically, the voice data is synchronized using a streaming protocol (e.g., WebRTC), and the movement data is synchronized using an AI animation engine (e.g., Live2D). The output is the synchronized voice and animated character data.

[1223] Step 7: Sending voice and animated characters

[1224] The server sends synchronized audio and animated character data to the client in real time. The input in this step is the synchronized audio and animated character data, specifically, the transmission uses WebSocket or other low-latency protocols. The output is the data sent to the client.

[1225] Step 8: Sending a Connection Request

[1226] The user uses a terminal to send a connection request to the call center through a dedicated application or browser. The input to this step is the user's connection request, specifically, using an HTTP request or WebSocket. The output is the connection request sent to the server.

[1227] Step 9: Select an operator

[1228] The list of waiting operators sent from the server is displayed on the terminal. The user selects the desired operator from the list and sends the selection information to the server. The input in this step is the list of waiting operators, and the output is the operator information selected by the user.

[1229] Step 10: Viewing Real-Time Sync

[1230] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal. The input is synchronized data transmitted from the server, specifically received via a low-latency protocol (e.g., WebRTC). The output is the animated character and voice displayed on the user's terminal.

[1231] Step 11: Record the interaction

[1232] Once the call is over, the user interaction is recorded. The input to this step is the data of the user interaction, specifically stored in the "interactions" table in the database. The output is the recorded interaction data.

[1233] (Application example 1)

[1234] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1235] Conventional call center systems rely mainly on voice communication, and lack visual interaction with users, resulting in lower user satisfaction compared to face-to-face support. Furthermore, there are also challenges such as the difficulty of providing prompt and friendly responses in emergencies and security-related inquiries.

[1236] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1237] In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the voice of the operator in real time, means for transmitting the synchronized voice and character to a user terminal, means for a user to select an operator, and means for applying the character display and voice synchronization to emergency calls and inquiries. This allows users to communicate with operators in real time in a visually friendly manner. This also allows for a rapid response in security-related emergencies.

[1238] "Operator" refers to a staff member who responds to user inquiries.

[1239] "Means for capturing audio" refers to a device or software for capturing the operator's speech in real time.

[1240] "Animated Character" refers to a dynamic character that is generated based on the operator's voice and visually displayed to the user.

[1241] "Real-time synchronization means" means any device or software that synchronizes the voice of an operator with the movements of an animated character.

[1242] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[1243] The term "means for a user to select an operator" refers to an interface or function that allows a user to select any operator from a plurality of operators.

[1244] "Applicable emergency communication and inquiry response measures" refers to the functions or processes that ensure efficient response to security-related emergencies and other inquiries.

[1245] The system for implementing this invention is composed of a server, a client terminal, and a user as its main components. The overall flow is that the server registers operator information, and based on that information, synchronizes the operator's voice with an animated character in real time and transmits it to the user terminal.

[1246] Server Configuration

[1247] The server consists of the following hardware and software:

[1248] Hardware: Server equipped with high-performance CPU, memory, storage, etc. (e.g. Dell PowerEdge)

[1249] Software: Python and Django for server-side processing, SQL for database management, TensorFlow and custom AI models for image generation and audio synchronization

[1250] function:

[1251] 1. Operator information registration:

[1252] The operator's name, profile picture and voice sample are entered into a database.

[1253] Using a voice sample, we request the image generation AI to generate an animated character.

[1254] The generated animated character is saved in a database.

[1255] 2. Waiting for client connections:

[1256] Waits for call center connection requests via telephone lines or the Internet.

[1257] 3. Operator selection and synchronization:

[1258] Receives requests from clients and sends a list of waiting operators to the user's terminal.

[1259] Real-time synchronization of voice and animated characters for selected operators.

[1260] 4. Voice and character transmission:

[1261] Synchronized voice and animated characters are transmitted to a user terminal in real time.

[1262] Client terminal

[1263] Client devices are primarily expected to be smartphones. Applications are developed and operated in the following environments:

[1264] Hardware: Android smartphone or iPhone

[1265] Software: Kotlin (for Android), Swift (for iOS), Unity (for animation display)

[1266] function:

[1267] 1. Send a connection request:

[1268] The user uses the application on the terminal to send a connection request to the call center.

[1269] 2. Operator Selection:

[1270] The list of waiting operators sent from the server is displayed on the terminal.

[1271] Provides an interface for users to select their preferred operator.

[1272] 3. Real-time Sync:

[1273] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[1274] 4. Recording interactions:

[1275] After the interaction, the user's interaction is recorded to collect feedback and satisfaction.

[1276] Specific examples

[1277] When a user detects an abnormality in their home security system during an emergency, they connect to the "Security Assistant" through the app. A list of operators is displayed, and the user can select one. The selected operator's animated character will appear in real time, and emergency response instructions will be given through synchronized voice.

[1278] Prompt Sentence Examples

[1279] Examples of prompts include:

[1280] "Please register your operator information. The operator's name is "Taro", the profile picture is the attached image, and the voice sample is the attached audio file. Please generate an animated character based on this information."

[1281] This configuration allows users to communicate visually with operators in real time and receive prompt and accurate responses, and also enables rapid response in security-related emergencies.

[1282] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1283] Step 1:

[1284] The server registers the operator's name, profile picture, and voice sample in the database. The input of this step is the operator's basic information, and the output is the operator's information stored in the database. Data processing involves analyzing the input voice sample, converting it into an appropriate format, and saving it.

[1285] Step 2:

[1286] The server uses the registered voice sample to request the image generation AI to generate an animated character. The input of this step is the voice sample, and the output is the generated animated character. Data calculation involves extracting voice features and inputting them into the image generation AI to generate the animated character.

[1287] Step 3:

[1288] The server saves the generated animated character in a database. The input of this step is the generated animated character, and the output is the character data saved in the database. Data processing involves converting the character data into an appropriate format and saving it.

[1289] Step 4:

[1290] The user uses the terminal to send a connection request to the call center. The input of this step is the user's connection request, and the output is the connection request information sent to the server. The specific operation is that the user opens the application and presses the call center connection button.

[1291] Step 5:

[1292] The server receives a call center connection request and sends a list of waiting operators to the user's terminal. The input of this step is the user's connection request, and the output is the list of waiting operators. Data processing involves converting the waiting operator information into a list format and sending it.

[1293] Step 6:

[1294] The user selects the desired operator from the list and sends the selection information to the server. The input of this step is the operator selection information, and the output is the selection information sent to the server. The specific operation is that the user taps on the terminal screen to select the operator.

[1295] Step 7:

[1296] The server synchronizes the voice of the selected operator with the animation character in real time. The input of this step is the operator's voice, and the output is the synchronized animation character. The data calculation is to analyze the voice data in real time and generate the animation movement accordingly.

[1297] Step 8:

[1298] The server transmits the synchronized voice and animated characters to the user terminal in real time. The input of this step is the synchronized voice and characters, and the output is the data transmitted to the user terminal. The data processing is to perform real-time data streaming.

[1299] Step 9:

[1300] The user interacts with the operator through an animated character displayed in real time on the terminal. The input of this step is the voice and character sent to the terminal, and the output is the user's interaction experience. The specific operation is for the user to continue the conversation with the operator while looking at the terminal screen.

[1301] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1302] The present invention relates to a system that combines real-time synchronization of an operator's voice with an animated character, and an emotion engine for recognizing the user's emotions. The system is implemented through the following steps:

[1303] server

[1304] 1. Register operator information

[1305] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[1306] The server sends a request to the image generation AI using the registered voice sample to generate an animated character of the operator.

[1307] The generated animated characters are stored in a database.

[1308] 2. Waiting for client connections

[1309] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[1310] The server maintains and updates a list of available operators.

[1311] 3. Operator selection and synchronization

[1312] The server receives a request from a client and sends a list of waiting operators to the client.

[1313] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[1314] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animated character.

[1315] 4. Emotion Recognition by Emotion Engine

[1316] The server uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[1317] The recognized emotion data is used to change the facial expressions and movements of the operator's animated character.

[1318] 5. Voice and character transmission

[1319] The server transmits synchronized audio and animated characters to the client in real time.

[1320] 6. Recording interaction and emotion data

[1321] The server records the user's interactions and the perceived emotion data and integrates it into subsequent feedback collection measures.

[1322] Terminal (client)

[1323] 1. Sending a connection request

[1324] Users use their devices to send connection requests to a call center, which can be made via telephone or a dedicated application.

[1325] 2. Operator Selection

[1326] The list of waiting operators sent from the server is displayed on the terminal.

[1327] The user selects a desired animated character from the list and transmits the selection information to the server.

[1328] 3. Real-time synchronization

[1329] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[1330] 4. Emotion-aware feedback

[1331] The terminal receives the user's emotion recognized by the emotion engine and provides an operator's response based on the emotion.

[1332] 5. Recording interactions

[1333] After the interaction, the user's interaction and emotional data is recorded and used to collect feedback and satisfaction.

[1334] User processing steps

[1335] 1. Connect to the call center

[1336] The user connects to the call center by phone or using an application.

[1337] 2. Confirm your operator selection

[1338] After connecting, check the list of waiting operators and select the character you want.

[1339] 3. Start of the conversation

[1340] The selected animated character will synchronize with the audio in real time and begin interacting.

[1341] 4. Providing emotional feedback

[1342] After the call is over, the emotional data recognized by the emotion engine is integrated into a feedback form to provide satisfaction and opinions.

[1343] Specific examples

[1344] Example of a system where Operator A is registered

[1345] Registering operator information

[1346] 1. The server registers Operator A's name, profile picture, and voice sample.

[1347] 2. The server uses operator A's voice sample to request the image generation AI to generate an animated character.

[1348] 3. The generated animated character is saved in the database.

[1349] User connects to call center

[1350] 1. A user connects to a call center using a phone or application.

[1351] 2. The server sends a list of waiting operators to the user terminal.

[1352] 3. The user selects an animated character for Operator A and sends that information back to the server.

[1353] Real-time synchronization begins

[1354] 1. The server captures operator A's voice in real time and synchronizes it with the corresponding animation character's movement data.

[1355] 2. Synchronized audio and animated characters are transmitted to the user's device in real time.

[1356] 3. The emotion engine recognizes the user's emotions and uses that data to change the facial expression of the operator's animated character.

[1357] 4. Users can receive service through the animated character as if they were face-to-face with Operator A.

[1358] This system allows users to enjoy a more emotional and face-to-face service compared to traditional call center services. Furthermore, by recording interactions and collecting feedback, it is possible to improve the quality of service.

[1359] The processing flow will be explained below.

[1360] Server Processing Steps

[1361] Step 1:

[1362] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[1363] Step 2:

[1364] The server sends the registered voice samples to an image generation AI to generate an animated character of the operator.

[1365] Step 3:

[1366] The server stores the generated animated characters in a database.

[1367] Step 4:

[1368] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[1369] Step 5:

[1370] The server maintains an updated list of available operators.

[1371] Step 6:

[1372] The server receives a connection request from a client and sends a list of waiting operators to the client.

[1373] Step 7:

[1374] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[1375] Step 8:

[1376] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animation character.

[1377] Step 9:

[1378] The server uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[1379] Step 10:

[1380] The server changes the facial expressions and movements of the operator's animated character based on the recognized emotional data.

[1381] Step 11:

[1382] The server transmits synchronized audio and animated characters to the client in real time.

[1383] Step 12:

[1384] The server records the user's interactions and the perceived emotion data and integrates it into subsequent feedback collection measures.

[1385] Terminal (client) processing steps

[1386] Step 1:

[1387] Users use their devices to send connection requests to a call center, which can be made via telephone or a dedicated application.

[1388] Step 2:

[1389] The list of waiting operators sent from the server is displayed on the terminal.

[1390] Step 3:

[1391] The user selects a desired animated character from the list and transmits the selection information to the server.

[1392] Step 4:

[1393] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[1394] Step 5:

[1395] The terminal receives the user's emotions recognized by the emotion engine and changes the facial expression of the operator's animated character in real time based on that information.

[1396] Step 6:

[1397] After answering the call, the terminal records the user's interaction and the recognized emotion data and transmits them to the server.

[1398] User processing steps

[1399] Step 1:

[1400] The user connects to the call center by phone or using an application.

[1401] Step 2:

[1402] After connecting, check the list of waiting operators and select the character you want.

[1403] Step 3:

[1404] The selected animated character will synchronize with the audio in real time and begin interacting.

[1405] Step 4:

[1406] Interact with an operator whose facial expressions change in real time thanks to an emotion engine.

[1407] Step 5:

[1408] After the call is over, the emotional data recognized by the emotion engine is integrated into a feedback form to provide satisfaction and opinions.

[1409] Through the above steps, the system can provide users with an emotional and face-to-face interaction, and the recording of interaction and emotional data provides valuable data for improving service quality.

[1410] Example 2

[1411] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1412] Conventional call center systems mainly rely on voice communication between operators and users, limiting interaction. Furthermore, they are unable to recognize users' emotions in real time and respond accordingly, resulting in a poor user experience. Furthermore, processes such as operator selection and interaction recording are inefficient, making it difficult to improve service quality. There is a need to solve these problems and provide responses that are more emotional and closer to face-to-face interactions.

[1413] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1414] In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the operator's voice in real time, means for transmitting the synchronized voice and character to a user terminal, means for the user to select an operator, means for recognizing the user's emotion, and means for changing the facial expressions and movements of the animated character based on the recognized emotion. This allows the user to receive emotional responses in real time, allowing for an interaction experience that is closer to face-to-face interaction. Furthermore, feedback can be collected based on interaction records and emotion data, making it possible to continuously improve the quality of service.

[1415] "Means for acquiring the operator's voice" refers to devices or software for capturing the voice data spoken by the operator in real time and inputting it into the system.

[1416] "Means for generating animated characters" means technologies and algorithms for generating animated characters that can move in real time based on operator voice and profile data.

[1417] "Means for synchronizing voice and characters in real time" refers to software or systems that allow for timely adjustment and reflection of the movements and expressions of animated characters based on the operator's voice data.

[1418] "Means for transmitting to the user terminal" refers to the communication technology and protocol for transmitting synchronized audio data and animation data from the server to the client terminal in real time.

[1419] "Means for the user to select an operator" refers to a GUI (graphical user interface) on the interface used by the user to select a desired operator from a plurality of operators.

[1420] "Means for recognizing user emotions" refers to an emotion recognition engine or algorithm that processes the user's voice data and facial expression data and analyzes their emotional state.

[1421] "Means for changing the facial expressions and movements of an animated character based on emotions" refers to a control system or algorithm that uses recognized user emotional data to change the facial expressions and movements of an animated character in real time.

[1422] "Means for storing operator information in a database" refers to the functions and systems for registering and storing basic information about operators (such as names, profile photos, and voice samples) in a database.

[1423] "Means for sending a list of waiting operators to a client" refers to a communication means or protocol for generating a list of waiting operators on the server side and sending it to the user's terminal.

[1424] "Means for recording interactions" refers to technologies and systems that record all voice interactions and actions between users and operators for later analysis and evaluation.

[1425] "Means for collecting feedback based on emotional data" refers to a method for analyzing emotional data collected during interactions with users and using the results to collect user satisfaction and areas for improvement through a feedback form, etc.

[1426] "Measures to improve service quality" refers to continuous improvement methods that analyze collected feedback data and use it to improve the way operators respond and the performance of the entire system.

[1427] MODE FOR CARRYING OUT THE INVENTION

[1428] This invention relates to a system that synchronizes voice and animation in real time and recognizes a user's emotions. This system is composed of a server, a terminal (client), and a user.

[1429] Basic Server Configuration

[1430] The server is the central component responsible for the main processing and implements the following functions:

[1431] 1. Register operator information

[1432] The server stores various information about the operator (name, profile photo, voice sample) in a database. An RDBMS such as MySQL is used here. Using Python's OpenAI API library, the voice sample is sent to a generative AI model (e.g., DALL-E) to generate an animated character. The generated character is then stored back in the database.

[1433] 2. Waiting for client connections

[1434] The server waits for and processes connection requests from clients using a communications platform such as Twilio.

[1435] 3. Operator selection and synchronization

[1436] Based on the client's request, the server sends a list of available operators to the client. When the user selects an operator, the server receives that information and synchronizes the operator's voice data with the animated character in real time. APIs such as Amazon Polly and Google Text-to-Speech are used for voice processing.

[1437] 4. Emotion Recognition by Emotion Engine

[1438] The server uses Azure Cognitive Services' Emotion API to recognize emotions from the user's voice and facial expressions, and the recognized emotional data is reflected in the facial expressions and movements of the animated character in real time.

[1439] 5. Voice and character transmission

[1440] The server sends synchronized audio and animated characters to the client using real-time communication such as WebSocket.

[1441] 6. Recording interaction and emotion data

[1442] The server records user interactions and emotional data and stores it in Amazon S3 or Google Cloud Storage, which helps gather subsequent feedback and improve the service.

[1443] Basic configuration of the terminal (client)

[1444] The client terminal is a device used by a user and has the following functions:

[1445] 1. Sending a connection request

[1446] Users use a dedicated smartphone app or web application to send a connection request to a server, which is established through a service such as Twilio.

[1447] 2. Operator Selection

[1448] The user checks the list of waiting operators sent from the server and selects the desired operator, and the selection information is sent to the server.

[1449] 3. Real-time synchronization

[1450] The client receives and displays the selected operator's voice and animated character in real time. The animated character moves in real time using the Unity engine or similar.

[1451] 4. Emotion-aware feedback

[1452] The client receives the user's emotion data recognized by the emotion engine and displays a response based on that data (such as changing the character's facial expression).

[1453] 5. Recording interactions

[1454] The client records the user's interaction and emotion data and sends it to the server, which uses the data to improve satisfaction and services.

[1455] Basic user actions

[1456] A user uses the system in the following steps:

[1457] 1. Connect to the call center

[1458] Users connect to the call center using a phone or an application, with the connection provided through a communications platform such as Twilio.

[1459] 2. Confirm your operator selection

[1460] The user checks the list of waiting operators presented by the server and selects a desired operator.

[1461] 3. Start of the conversation

[1462] The selected animated character will be synchronized with the voice in real time, allowing the user to receive a face-to-face response, and the emotion engine will recognize the user's emotions in real time, providing a more emotionally relevant response.

[1463] 4. Providing emotional feedback

[1464] After the call is completed, a feedback form is displayed based on the emotional data recognized by the emotion engine, allowing the user to provide their satisfaction and opinions.

[1465] Specific examples

[1466] Prompt Sentence Examples

[1467] Example: Use the voice sample of Operator A to generate an animated character using a generative AI model. Voice sample URL: http: / / example.com / audio_sample

[1468] Compared to traditional call centers, the system offers a more empathetic, face-to-face experience, and by recording interactions and collecting feedback, it can continuously improve service quality.

[1469] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1470] Program processing flow

[1471] server

[1472] Step 1: Register operator information

[1473] Input: Operator name, profile picture, voice sample

[1474] What it does: The server receives the operator's name, profile picture, and voice sample from the form and stores the data in a MySQL database.

[1475] Data processing: The server uses Python's OpenAI API library to send voice samples to DALL-E with prompt sentences to generate a corresponding animated character.

[1476] Output: The generated animated characters are saved back to the database.

[1477] Step 2: Waiting for client connections

[1478] Input: Connection request from the client

[1479] Specific operation: The server waits for a call center connection request from a client using a cloud communication platform such as Twilio.

[1480] Data Processing: Manage and update the list of available operators in real time.

[1481] Output: Generates the operator list upon receiving a request from the client.

[1482] Step 3: Select and sync operators

[1483] Input: Operator selection information from the user

[1484] Specific operation: Based on a request from a client, the server sends a list of waiting operators to the client.

[1485] Data processing: The voice of the selected operator is synchronized with the movement data of the corresponding animated character in real time. Amazon Polly is used for voice processing.

[1486] Output: Synchronized data is generated.

[1487] Step 4: Emotion Recognition with the Emotion Engine

[1488] Input: User's voice and facial expressions

[1489] Specific operation: The server uses the Emotion API of Azure Cognitive Services to recognize emotions from the user's voice and facial expressions.

[1490] Data processing: Collect recognized emotion data and update the facial expressions and movements of the animated character.

[1491] Output: Updated animated character data is generated.

[1492] Step 5: Submit your voice and character

[1493] Input: Synchronized audio and animated character data

[1494] Specific operation: The server transmits synchronized audio and animated characters to the client in real time using WebSocket.

[1495] Data processing: Convert the data into a WebSocket communication format.

[1496] Output: Generates data sent to the client.

[1497] Step 6: Recording interaction and emotion data

[1498] Input: User interaction data and recognized emotion data

[1499] Specific Operation: The server records the user's interactions and the recognized emotion data.

[1500] Data processing: The recorded data is stored on Amazon S3 or Google Cloud Storage.

[1501] Output: The saved data is generated.

[1502] Terminal (client)

[1503] Step 1: Sending a Connection Request

[1504] Input: A user requesting a connection

[1505] How it works: A user sends a connection request to a server using a dedicated smartphone app or web application. This connection is made through a service such as Twilio.

[1506] Data processing: Generate connection request data.

[1507] Output: Generates the connection request data sent to the server.

[1508] Step 2: Select an operator

[1509] Input: List of waiting operators from the server

[1510] Specific operation: The user checks the list of waiting operators sent from the server and selects the desired operator. The selection information is sent to the server.

[1511] Data processing: Generate user selection information.

[1512] Output: Selection information is generated that is sent to the server.

[1513] Step 3: Real-time sync

[1514] Input: Synchronized audio and animated character data sent from the server

[1515] Specific operation: The client receives and displays the selected operator's voice and animated character in real time. The animated character moves in real time using the Unity engine or similar.

[1516] Data processing: Converting received data into a playable format.

[1517] Output: Synchronized voice and character data is generated that is displayed to the user.

[1518] Step 4: Emotion recognition feedback

[1519] Input: Emotion data from the server

[1520] Specific operation: The client receives the user emotion data recognized by the emotion recognition engine and displays a response based on that data (such as changing the character's facial expression).

[1521] Data processing: Generate animation update data based on emotion data.

[1522] Output: Updated animated character data is generated that is displayed to the user.

[1523] Step 5: Record the interaction

[1524] Input: User interaction data and emotion data

[1525] Specific operation: After the interaction, the client records the user's interaction data and emotion data and sends them to the server.

[1526] Data processing: Generate recorded interaction data and emotion data.

[1527] Output: Interaction data and emotion data are generated and sent to the server.

[1528] User

[1529] Step 1: Connect to a call center

[1530] Input: A user requesting a connection

[1531] What it does: A user connects to a call center using a phone or a dedicated app, and the connection is made through a communications platform such as Twilio.

[1532] Data processing: Generate connection request data and send it to the server.

[1533] Output: The connection request data sent to the server is generated.

[1534] Step 2: Confirm operator selection

[1535] Input: List of waiting operators from the server

[1536] Specific operation: The user checks the list of waiting operators presented by the server and selects the desired operator.

[1537] Data processing: Generate user selection information and send it to the server.

[1538] Output: Selection information is generated that is sent to the server.

[1539] Step 3: Start the conversation

[1540] Input: Synchronized audio and animated character data sent from the server

[1541] Specific operation: An animated character selected by the user is synchronized with the audio in real time, and the user is treated as if they were face-to-face.

[1542] Data processing: Converting received audio and animation data into a playable format.

[1543] Output: Synchronized voice and character data is generated that is displayed to the user.

[1544] Step 4: Provide emotional feedback

[1545] Input: Emotion data from the server

[1546] Specific operation: Based on the emotion data recognized by the emotion engine, a feedback form is displayed on the user's device. The user provides their satisfaction and opinions.

[1547] Data processing: Generate user feedback data and send it to the server.

[1548] Output: Feedback data is generated that is sent to the server.

[1549] (Application example 2)

[1550] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1551] Conventional customer service systems have the problem that the operator's facial expressions and emotions are difficult to convey, making communication with the user feel mechanical and cold. Furthermore, the lack of a system that can flexibly respond to the user's choices and emotions makes it difficult to provide highly satisfying service. This is particularly true in brick-and-mortar stores, where it is difficult for operators in remote locations to provide effective customer service in real time.

[1552] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the voice of the operator in real time, means for transmitting the synchronized voice and character to a user terminal, means for the user to select an operator, means for recognizing the user's emotion, and means for changing the facial expression of the animated character based on the recognized emotion. This allows the user to enjoy an emotional response similar to that of a face-to-face interaction, thereby enabling highly satisfying customer service.

[1553] "Means for acquiring the operator's voice" means a device or software for receiving the voice signal emitted by the operator and recording and processing it as digital data.

[1554] "Means for generating animated characters" means algorithms or software for automatically creating 2D or 3D animated characters based on operator voice or other data.

[1555] "Means of real-time synchronization" refers to technology or systems that instantly match the operator's voice and the movements of the animated character, displaying them without any time lag.

[1556] The "means for transmitting to the user terminal" is a communication function for transmitting the generated animated character and the audio synchronized therewith to the user's device via a network.

[1557] "Means for a user to select an operator" refers to an interface or system that allows a user to select a specific operator from among multiple operators.

[1558] "Means for recognizing user emotions" refers to algorithms and implementations for analyzing and identifying the emotional state of a user from their facial expressions and voice.

[1559] "Means for changing the facial expression of an animated character" refers to a function or system for appropriately changing the facial expression or movement of an animated character in accordance with the recognized emotions of the user.

[1560] "Means for storing operator information in a database" refers to a database system that centrally manages and stores basic information about operators, voice samples, generated character information, etc.

[1561] The "means for transmitting a list of waiting operators to a client" refers to a function or technology for transmitting a list of currently available operators to a user's device.

[1562] The "means for generating data for simulating information about a product" is a simulation function for automatically generating data required for product explanations and demonstrations.

[1563] The "means for providing product information to users" refers to a display function or interface for effectively conveying the generated product data to users.

[1564] "Means for recording user interactions" refers to a system for saving the content of conversations and actions with users as digital data.

[1565] "Means of collecting feedback" are surveys and rating systems for collecting user satisfaction and opinions.

[1566] "Means for dynamically changing customer service content based on user interaction" refers to algorithms and technologies that adapt and change customer service content in real time according to the user's dialogue and behavior.

[1567] The present invention relates to a system for recognizing a user's emotions by synchronizing an operator's voice with an animated character in real time. A specific embodiment of this system will now be described.

[1568] server

[1569] The server first captures the operator's voice. Commonly used hardware includes a microphone to record the voice as digital data and a server computer to process it. The voice data is captured using digital signal processing (DSP) technology.

[1570] The server then generates an animated character based on the voice data. The software used is a generative AI model (e.g., OpenAI's DALL-E or MidJourney). It analyzes the voice sample and generates an animated character based on its characteristics. This generated character is then stored in a database.

[1571] The server analyzes the voice data and links it with the character's movement data to synchronize the generated animated character and the operator's voice in real time. This synchronization is achieved using real-time voice analysis software and motion capture technology.

[1572] The server has a communication function to transmit synchronized audio and animated characters to the user terminal, using the commonly used Internet Protocol (TCP / IP) as the communication protocol.

[1573] The server also recognizes the user's emotions and dynamically changes the facial expression of the animated character based on that data. This emotion recognition uses the Hugging Face transformers library and DeepFace. The server analyzes emotions from the user's voice and facial expressions, dynamically changing the character's facial expression accordingly.

[1574] In addition, the server stores operator information in a database and has the function of sending a list of waiting operators to the user's terminal. It also plays a role in improving the efficiency of customer service by generating data to simulate product information and providing it to the user.

[1575] Terminal (client)

[1576] The user terminals include smartphones and smart glasses, and connect to the call center using these terminals. The user can send a connection request through the application and check the list of waiting operators. This list is provided by the server, and the user can select the operator they want.

[1577] Once the selection is complete, the voice and animated character are synchronized from the server and displayed in real time on the user's device. The device is equipped with a camera and microphone to recognize the user's emotions. This emotional data is sent to the server, and the character's facial expression changes dynamically based on the content of the conversation.

[1578] User

[1579] Users interact with the virtual store clerk through the terminal, and the system also has the ability to record user interactions and collect feedback, which is used to improve subsequent services.

[1580] Specifically, when a user visits a store and launches the app on their smartphone, a list of available virtual clerks is displayed. The user selects a virtual clerk from the list and begins a conversation. The system then recognizes the user's emotions and changes the virtual clerk's response accordingly. The following is an example of a prompt sentence:

[1581] Example prompt sentence:

[1582] "The user launches the app and gets a list of available virtual sales associates. The user selects a virtual sales associate and begins interacting with them."

[1583] In this way, users can enjoy a more emotional and face-to-face interaction, making it possible to provide a high level of satisfaction that was difficult to achieve with conventional customer service systems.

[1584] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1585] Step 1:

[1586] The server captures the operator's voice. Specifically, when the operator speaks into the microphone, the voice is sent to the server as a digital signal. This voice data is analyzed using digital signal processing (DSP) and saved in a database in real time. The input is the operator's voice through the microphone, and the output is the analyzed digital voice data.

[1587] Step 2:

[1588] The server generates an animated character based on the operator's voice data. It uses a generative AI model (e.g., DALL-E or MidJourney) to generate an image of the animated character from the voice features. This generated character is stored in a database. The input is the operator's voice sample, and the output is an image of the generated animated character.

[1589] Step 3:

[1590] The server synchronizes the generated animated character with the operator's voice in real time. To do this, it uses real-time voice analysis software to analyze the voice data and match the character's mouth movements and facial expressions. The input is digital voice data and an image of the animated character, and the output is a synchronized set of animated character and voice.

[1591] Step 4:

[1592] The server transmits synchronized audio and animated characters to the user device. The communication protocol is HTTP or WebSocket, and data is streamed to the user device in real time. The input is synchronized animated character and audio data, and the output is real-time animation displayed on the user device.

[1593] Step 5:

[1594] The terminal sends a connection request from the user to the server. Specifically, when the user launches an application on their smartphone or smart glasses and presses the connect button, the request is sent to the server. The input is the user's connection request, and the output is the connection request data sent to the server.

[1595] Step 6:

[1596] The server sends a list of waiting operators to the user terminal. It retrieves operator information from the database, generates a list of available operators, and sends it to the user terminal. The input is the operator information stored in the database, and the output is the list of waiting operators displayed on the user terminal.

[1597] Step 7:

[1598] The user selects an operator and sends the selected information to the server through the terminal. The input is the user's selection operation, and the output is the selected information sent to the server.

[1599] Step 8:

[1600] The server analyzes the user's facial expression data received from the device to recognize the user's emotions. DeepFace is used for emotion recognition, and the analysis results are reflected in the character's facial expressions. The input is the user's facial expression data, and the output is the recognized emotional data and the character's facial expressions that change based on it.

[1601] Step 9:

[1602] The server records interactions and collects feedback. The content of the dialogue with the user is saved as digital data and later analyzed. This data is analyzed by the feedback collection system and used to improve the service. The input is the dialogue data with the user, and the output is the analysis results as feedback.

[1603] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1604] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1605] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1606] [Fourth embodiment]

[1607] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1608] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1609] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1610] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1611] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1612] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1613] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1614] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1615] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1616] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1617] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1618] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1619] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1620] The system according to the present invention synchronizes the voice of an operator with an animated character in real time, providing users with a more face-to-face interaction. This system can be implemented through the following steps:

[1621] server

[1622] 1. Register operator information

[1623] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[1624] The server sends a request to the image generation AI using the registered voice sample to generate an animated character of the operator.

[1625] The generated animated character is again saved in the database.

[1626] 2. Waiting for client connections

[1627] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[1628] The server maintains and updates a list of available operators.

[1629] 3. Operator selection and synchronization

[1630] When the server receives a request from a client, it sends a list of waiting operators to the client.

[1631] Once the client selects the desired operator, the information is returned to the server, which synchronizes the voice and animated character of the selected operator in real time.

[1632] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animated character.

[1633] 4. Sending voice and characters

[1634] The server transmits synchronized audio and animated characters to the client in real time.

[1635] Terminal (client)

[1636] 1. Sending a connection request

[1637] Users use their devices to send connection requests to a call center, which can be made by phone or through a dedicated application.

[1638] 2. Operator Selection

[1639] The list of waiting operators sent from the server is displayed on the terminal.

[1640] The user selects a desired animated character from the list and transmits the selection information to the server.

[1641] 3. Real-time synchronization

[1642] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[1643] 4. Recording interactions

[1644] After the interaction, the user's interaction is recorded and used to collect feedback and satisfaction.

[1645] Specific examples

[1646] Example of a system where Operator A is registered

[1647] Registering operator information

[1648] 1. The server registers Operator A's name, profile picture, and voice sample.

[1649] 2. The server uses operator A's voice sample to request the image generation AI to generate an animated character.

[1650] 3. The generated animated character is saved in the database.

[1651] User connects to call center

[1652] 1. A user connects to a call center using a phone or application.

[1653] 2. The server sends a list of waiting operators to the user terminal.

[1654] 3. The user selects an animated character for Operator A and sends that information back to the server.

[1655] Real-time synchronization begins

[1656] 1. The server captures operator A's voice in real time and synchronizes it with the movement data of the animation character that corresponds to that voice.

[1657] 2. Synchronized audio and animated characters are transmitted to the user's device in real time.

[1658] 3. Users can receive service as if they were face-to-face with Operator A through the animated character.

[1659] This system allows users to enjoy a more emotional and face-to-face service compared to traditional call center services. Furthermore, by recording interactions and collecting feedback, it is possible to improve the quality of service.

[1660] The processing flow will be explained below.

[1661] Server Processing Steps

[1662] Step 1:

[1663] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[1664] Step 2:

[1665] The server sends the voice samples to an image generation AI to generate an animated character for the operator.

[1666] Step 3:

[1667] The server stores the generated animated characters in a database.

[1668] Step 4:

[1669] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[1670] Step 5:

[1671] The server maintains an updated list of available operators.

[1672] Step 6:

[1673] The server receives a connection request from a client and sends a list of waiting operators to the client.

[1674] Step 7:

[1675] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[1676] Step 8:

[1677] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animation character.

[1678] Step 9:

[1679] The server transmits synchronized audio and animated characters to the client in real time.

[1680] Terminal (client) processing steps

[1681] Step 1:

[1682] The user uses a terminal to send a connection request to the call center, which can be made over the phone or through an application.

[1683] Step 2:

[1684] The list of waiting operators sent from the server is displayed on the terminal.

[1685] Step 3:

[1686] The user selects a desired animated character from the list and transmits the selection information to the server.

[1687] Step 4:

[1688] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[1689] Step 5:

[1690] After the interaction, the user's interaction is recorded and used to collect feedback and satisfaction.

[1691] User processing steps

[1692] Step 1:

[1693] The user connects to the call center by phone or using an application.

[1694] Step 2:

[1695] After connecting, check the list of waiting operators and select the character you want.

[1696] Step 3:

[1697] The selected animated character will synchronize with the audio in real time and begin interacting.

[1698] Step 4:

[1699] After the interaction, use the feedback form to provide your satisfaction and opinions.

[1700] Example 1

[1701] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1702] In conventional call center systems, users communicate with operators only through voice, which means there is a lack of visual information, making it difficult to convey emotions and facial expressions compared to face-to-face interactions. Furthermore, the limited options for operators can lead to lower user satisfaction. Furthermore, the lack of interaction recording and feedback collection functions poses a challenge, resulting in a lack of data to improve service quality.

[1703] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1704] In this invention, the server includes a means for acquiring the voice of an operator, a means for generating a prompt sentence for a generative AI model based on the voice and generating an animated character, a means for synchronizing the generated animated character with the voice of the operator in real time, a means for transmitting the synchronized voice and character to the client, and a means for the user to select an animated character for the operator. This allows users to enjoy a visually rich, face-to-face interaction, and the increased choice of operators increases satisfaction. Furthermore, by recording interactions and collecting feedback, the quality of service can be continuously improved.

[1705] "Operator" means a person who provides services and support to users via telecommunications.

[1706] "Voice" refers to data of the voice or speech produced by the operator.

[1707] A "generative AI model" is an artificial intelligence algorithm or program that generates new data based on input data. Specifically, it refers to a model for generating animated characters from voice or text.

[1708] A "prompt" is text that is input to a generative AI model and contains instructions and conditions for obtaining a specific output.

[1709] An "animated character" is a virtual figure or character generated based on the operator's voice and movements.

[1710] "Real-time" means a state in which delays are extremely small and immediate responses are possible.

[1711] "Synchronization" means moving or matching multiple pieces of data or actions simultaneously.

[1712] A "client" refers to a terminal or software for using the system, and is connected to the server primarily to process user requests.

[1713] A "database" is a collection of information organized to efficiently store, retrieve, and update data.

[1714] "Interaction" is the process of exchange or operation between a user and a system or operator.

[1715] "Feedback" refers to the evaluations and opinions provided by users regarding the quality and usability of the Service, and includes the process of improving the Service based on the collected information.

[1716] The system according to the present invention synchronizes the voice of an operator with an animated character in real time, providing users with a service that is almost like a face-to-face interaction. This system is configured as follows.

[1717] server

[1718] The server registers the operator's basic information (name, profile photo, voice sample) in a database. After registration, the voice sample is used to generate a prompt for a generative AI model (e.g., DALL-E or Stable Diffusion) to generate an animated character. Specifically, an example of a prompt might be, "Based on this voice sample, please generate a real-time animated character for Operator A." The generated animated character is saved in the database.

[1719] The server waits for a connection request from the user. Once a connection is made, it sends a list of available operators to the client. Once the user selects an operator, it uses this information to synchronize the voice and animated characters in real time and transmits them to the client using a low-latency protocol (e.g., WebRTC).

[1720] Terminal (client)

[1721] The user uses their device to send a connection request to the call center through a dedicated application or browser. Once connected, a list of available operators sent from the server is displayed, and the user selects the desired operator. The selection information is sent back to the server, and real-time voice and animated characters are displayed on the device. The necessary data is received via a low-latency protocol such as WebRTC and rendered in real time.

[1722] User

[1723] Users can easily connect to the call center using a dedicated application or browser and select an operator. After the call is completed, the interaction is recorded and users can provide feedback to improve the quality of the service. This interaction and feedback are stored in a database and used to improve the service.

[1724] Specific examples

[1725] An example will be explained in which operator A is newly registered.

[1726] 1. Register operator information

[1727] The server registers Operator A's name, profile picture and voice sample in a database.

[1728] The server uses the voice sample to generate a prompt for the generation AI (e.g., Stable Diffusion), specifically, the prompt sentence "Based on this voice sample, please generate a real-time animated character of Operator A."

[1729] The generated animated characters are stored in a database.

[1730] 2. The user connects to the call center

[1731] Users use a dedicated application to connect to the call center by sending an HTTP request.

[1732] The server sends a list of waiting operators to the user terminal.

[1733] The user selects the desired operator and returns the selection information to the server.

[1734] 3. Start syncing in real time

[1735] The server captures the audio data in real time and synchronizes it with the animated character using an AI animation engine (e.g., Live2D).

[1736] The server sends the synchronized data to the client using WebRTC.

[1737] The user can view the transmitted audio and animated characters in real time using a dedicated application.

[1738] In this way, the system of the present invention allows users to enjoy visually rich, face-to-face interaction, improving the quality of service.

[1739] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1740] Step 1: Register operator information

[1741] The server registers basic information of the operator (name, profile photo, voice sample) in the database. As input, it receives the operator's name, profile photo, and voice sample. In concrete terms, it collects the voice sample and stores it in the "operators" table of the database along with the name and photo. The output is the operator information registered in the database.

[1742] Step 2: Sending a request to the image generation AI

[1743] The server uses the registered voice sample to generate a prompt for a generative AI model (e.g., DALL-E or Stable Diffusion) and generates an animated character. The input in this step is the voice sample, and an example prompt is "Please generate a real-time animated character for Operator A based on this voice sample." Specifically, the server sends a prompt to the generative AI model to generate an animated character. The output is data for the generated animated character, which is stored in a database.

[1744] Step 3: Waiting for client connections

[1745] The server waits for connection requests from users. The input includes user connection requests, which may come over telephone lines or the Internet. Specific operations include listening on a TCP / IP port or using a telephony API. The output is a list of connected clients.

[1746] Step 4: Submit the Operator List

[1747] When the server receives a connection request, it sends a list of waiting operators to the client. The input in this step is the waiting operator information retrieved from the database, and this information is sent to the client in JSON format. The output is the operator list displayed on the user's terminal.

[1748] Step 5: Receive operator selection information

[1749] When the user selects the desired operator, the selection information is sent back to the server. The input of this step is the operator information selected by the user, specifically, the selection information is sent in JSON format. The output is the selection information received by the server.

[1750] Step 6: Synchronizing the audio with the animated character

[1751] The server captures the selected operator's voice in real time and synchronizes it with the corresponding animated character's movement data. The input in this step is real-time voice data and the generated animated character's movement data. Specifically, the voice data is synchronized using a streaming protocol (e.g., WebRTC), and the movement data is synchronized using an AI animation engine (e.g., Live2D). The output is the synchronized voice and animated character data.

[1752] Step 7: Sending voice and animated characters

[1753] The server sends synchronized audio and animated character data to the client in real time. The input in this step is the synchronized audio and animated character data, specifically, the transmission uses WebSocket or other low-latency protocols. The output is the data sent to the client.

[1754] Step 8: Sending a Connection Request

[1755] The user uses a terminal to send a connection request to the call center through a dedicated application or browser. The input to this step is the user's connection request, specifically, using an HTTP request or WebSocket. The output is the connection request sent to the server.

[1756] Step 9: Select an operator

[1757] The list of waiting operators sent from the server is displayed on the terminal. The user selects the desired operator from the list and sends the selection information to the server. The input in this step is the list of waiting operators, and the output is the operator information selected by the user.

[1758] Step 10: Viewing Real-Time Sync

[1759] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal. The input is synchronized data transmitted from the server, specifically received via a low-latency protocol (e.g., WebRTC). The output is the animated character and voice displayed on the user's terminal.

[1760] Step 11: Record the interaction

[1761] Once the call is over, the user interaction is recorded. The input to this step is the data of the user interaction, specifically stored in the "interactions" table in the database. The output is the recorded interaction data.

[1762] (Application example 1)

[1763] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1764] Conventional call center systems rely mainly on voice communication, and lack visual interaction with users, resulting in lower user satisfaction compared to face-to-face support. Furthermore, there are also challenges such as the difficulty of providing prompt and friendly responses in emergencies and security-related inquiries.

[1765] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1766] In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the voice of the operator in real time, means for transmitting the synchronized voice and character to a user terminal, means for a user to select an operator, and means for applying the character display and voice synchronization to emergency calls and inquiries. This allows users to communicate with operators in real time in a visually friendly manner. This also allows for a rapid response in security-related emergencies.

[1767] "Operator" refers to a staff member who responds to user inquiries.

[1768] "Means for capturing audio" refers to a device or software for capturing the operator's speech in real time.

[1769] "Animated Character" refers to a dynamic character that is generated based on the operator's voice and visually displayed to the user.

[1770] "Real-time synchronization means" means any device or software that synchronizes the voice of an operator with the movements of an animated character.

[1771] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[1772] The term "means for a user to select an operator" refers to an interface or function that allows a user to select any operator from a plurality of operators.

[1773] "Applicable emergency communication and inquiry response measures" refers to the functions or processes that ensure efficient response to security-related emergencies and other inquiries.

[1774] The system for implementing this invention is composed of a server, a client terminal, and a user as its main components. The overall flow is that the server registers operator information, and based on that information, synchronizes the operator's voice with an animated character in real time and transmits it to the user terminal.

[1775] Server Configuration

[1776] The server consists of the following hardware and software:

[1777] Hardware: Server equipped with high-performance CPU, memory, storage, etc. (e.g. Dell PowerEdge)

[1778] Software: Python and Django for server-side processing, SQL for database management, TensorFlow and custom AI models for image generation and audio synchronization

[1779] function:

[1780] 1. Operator information registration:

[1781] The operator's name, profile picture and voice sample are entered into a database.

[1782] Using a voice sample, we request the image generation AI to generate an animated character.

[1783] The generated animated character is saved in a database.

[1784] 2. Waiting for client connections:

[1785] Waits for call center connection requests via telephone lines or the Internet.

[1786] 3. Operator selection and synchronization:

[1787] Receives requests from clients and sends a list of waiting operators to the user's terminal.

[1788] Real-time synchronization of voice and animated characters for selected operators.

[1789] 4. Voice and character transmission:

[1790] Synchronized voice and animated characters are transmitted to a user terminal in real time.

[1791] Client terminal

[1792] Client devices are primarily expected to be smartphones. Applications are developed and operated in the following environments:

[1793] Hardware: Android smartphone or iPhone

[1794] Software: Kotlin (for Android), Swift (for iOS), Unity (for animation display)

[1795] function:

[1796] 1. Send a connection request:

[1797] The user uses the application on the terminal to send a connection request to the call center.

[1798] 2. Operator Selection:

[1799] The list of waiting operators sent from the server is displayed on the terminal.

[1800] Provides an interface for users to select their preferred operator.

[1801] 3. Real-time Sync:

[1802] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[1803] 4. Recording interactions:

[1804] After the interaction, the user's interaction is recorded to collect feedback and satisfaction.

[1805] Specific examples

[1806] When a user detects an abnormality in their home security system during an emergency, they connect to the "Security Assistant" through the app. A list of operators is displayed, and the user can select one. The selected operator's animated character will appear in real time, and emergency response instructions will be given through synchronized voice.

[1807] Prompt Sentence Examples

[1808] Examples of prompts include:

[1809] "Please register your operator information. The operator's name is "Taro", the profile picture is the attached image, and the voice sample is the attached audio file. Please generate an animated character based on this information."

[1810] This configuration allows users to communicate visually with operators in real time and receive prompt and accurate responses, and also enables rapid response in security-related emergencies.

[1811] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1812] Step 1:

[1813] The server registers the operator's name, profile picture, and voice sample in the database. The input of this step is the operator's basic information, and the output is the operator's information stored in the database. Data processing involves analyzing the input voice sample, converting it into an appropriate format, and saving it.

[1814] Step 2:

[1815] The server uses the registered voice sample to request the image generation AI to generate an animated character. The input of this step is the voice sample, and the output is the generated animated character. Data calculation involves extracting voice features and inputting them into the image generation AI to generate the animated character.

[1816] Step 3:

[1817] The server saves the generated animated character in a database. The input of this step is the generated animated character, and the output is the character data saved in the database. Data processing involves converting the character data into an appropriate format and saving it.

[1818] Step 4:

[1819] The user uses the terminal to send a connection request to the call center. The input of this step is the user's connection request, and the output is the connection request information sent to the server. The specific operation is that the user opens the application and presses the call center connection button.

[1820] Step 5:

[1821] The server receives a call center connection request and sends a list of waiting operators to the user's terminal. The input of this step is the user's connection request, and the output is the list of waiting operators. Data processing involves converting the waiting operator information into a list format and sending it.

[1822] Step 6:

[1823] The user selects the desired operator from the list and sends the selection information to the server. The input of this step is the operator selection information, and the output is the selection information sent to the server. The specific operation is that the user taps on the terminal screen to select the operator.

[1824] Step 7:

[1825] The server synchronizes the voice of the selected operator with the animation character in real time. The input of this step is the operator's voice, and the output is the synchronized animation character. The data calculation is to analyze the voice data in real time and generate the animation movement accordingly.

[1826] Step 8:

[1827] The server transmits the synchronized voice and animated characters to the user terminal in real time. The input of this step is the synchronized voice and characters, and the output is the data transmitted to the user terminal. The data processing is to perform real-time data streaming.

[1828] Step 9:

[1829] The user interacts with the operator through an animated character displayed in real time on the terminal. The input of this step is the voice and character sent to the terminal, and the output is the user's interaction experience. The specific operation is for the user to continue the conversation with the operator while looking at the terminal screen.

[1830] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1831] The present invention relates to a system that combines real-time synchronization of an operator's voice with an animated character, and an emotion engine for recognizing the user's emotions. The system is implemented through the following steps:

[1832] server

[1833] 1. Register operator information

[1834] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[1835] The server sends a request to the image generation AI using the registered voice sample to generate an animated character of the operator.

[1836] The generated animated characters are stored in a database.

[1837] 2. Waiting for client connections

[1838] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[1839] The server maintains and updates a list of available operators.

[1840] 3. Operator selection and synchronization

[1841] The server receives a request from a client and sends a list of waiting operators to the client.

[1842] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[1843] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animated character.

[1844] 4. Emotion Recognition by Emotion Engine

[1845] The server uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[1846] The recognized emotion data is used to change the facial expressions and movements of the operator's animated character.

[1847] 5. Voice and character transmission

[1848] The server transmits synchronized audio and animated characters to the client in real time.

[1849] 6. Recording interaction and emotion data

[1850] The server records the user's interactions and the perceived emotion data and integrates it into subsequent feedback collection measures.

[1851] Terminal (client)

[1852] 1. Sending a connection request

[1853] Users use their devices to send connection requests to a call center, which can be made via telephone or a dedicated application.

[1854] 2. Operator Selection

[1855] The list of waiting operators sent from the server is displayed on the terminal.

[1856] The user selects a desired animated character from the list and transmits the selection information to the server.

[1857] 3. Real-time synchronization

[1858] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[1859] 4. Emotion-aware feedback

[1860] The terminal receives the user's emotion recognized by the emotion engine and provides an operator's response based on the emotion.

[1861] 5. Recording interactions

[1862] After the interaction, the user's interaction and emotional data is recorded and used to collect feedback and satisfaction.

[1863] User processing steps

[1864] 1. Connect to the call center

[1865] The user connects to the call center by phone or using an application.

[1866] 2. Confirm your operator selection

[1867] After connecting, check the list of waiting operators and select the character you want.

[1868] 3. Start of the conversation

[1869] The selected animated character will synchronize with the audio in real time and begin interacting.

[1870] 4. Providing emotional feedback

[1871] After the call is over, the emotional data recognized by the emotion engine is integrated into a feedback form to provide satisfaction and opinions.

[1872] Specific examples

[1873] Example of a system where Operator A is registered

[1874] Registering operator information

[1875] 1. The server registers Operator A's name, profile picture, and voice sample.

[1876] 2. The server uses operator A's voice sample to request the image generation AI to generate an animated character.

[1877] 3. The generated animated character is saved in the database.

[1878] User connects to call center

[1879] 1. A user connects to a call center using a phone or application.

[1880] 2. The server sends a list of waiting operators to the user terminal.

[1881] 3. The user selects an animated character for Operator A and sends that information back to the server.

[1882] Real-time synchronization begins

[1883] 1. The server captures operator A's voice in real time and synchronizes it with the corresponding animation character's movement data.

[1884] 2. Synchronized audio and animated characters are transmitted to the user's device in real time.

[1885] 3. The emotion engine recognizes the user's emotions and uses that data to change the facial expression of the operator's animated character.

[1886] 4. Users can receive service through the animated character as if they were face-to-face with Operator A.

[1887] This system allows users to enjoy a more emotional and face-to-face service compared to traditional call center services. Furthermore, by recording interactions and collecting feedback, it is possible to improve the quality of service.

[1888] The processing flow will be explained below.

[1889] Server Processing Steps

[1890] Step 1:

[1891] The server registers the operator's basic information (name, profile picture, voice sample) in a database.

[1892] Step 2:

[1893] The server sends the registered voice samples to an image generation AI to generate an animated character of the operator.

[1894] Step 3:

[1895] The server stores the generated animated characters in a database.

[1896] Step 4:

[1897] The server waits for requests to connect to the call center, which can be made over the phone line or over the Internet.

[1898] Step 5:

[1899] The server maintains an updated list of available operators.

[1900] Step 6:

[1901] The server receives a connection request from a client and sends a list of waiting operators to the client.

[1902] Step 7:

[1903] When a desired operator is selected by the client, the selection information is received by the server, and the voice of the selected operator is synchronized with the animation character in real time.

[1904] Step 8:

[1905] The server captures the operator's voice data in real time and synchronizes it with the corresponding movement data of the animation character.

[1906] Step 9:

[1907] The server uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[1908] Step 10:

[1909] The server changes the facial expressions and movements of the operator's animated character based on the recognized emotional data.

[1910] Step 11:

[1911] The server transmits synchronized audio and animated characters to the client in real time.

[1912] Step 12:

[1913] The server records the user's interactions and the perceived emotion data and integrates it into subsequent feedback collection measures.

[1914] Terminal (client) processing steps

[1915] Step 1:

[1916] Users use their devices to send connection requests to a call center, which can be made via telephone or a dedicated application.

[1917] Step 2:

[1918] The list of waiting operators sent from the server is displayed on the terminal.

[1919] Step 3:

[1920] The user selects a desired animated character from the list and transmits the selection information to the server.

[1921] Step 4:

[1922] The voice and animated character of the selected operator are transmitted from the server and displayed in real time on the terminal.

[1923] Step 5:

[1924] The terminal receives the user's emotions recognized by the emotion engine and changes the facial expression of the operator's animated character in real time based on that information.

[1925] Step 6:

[1926] After answering the call, the terminal records the user's interaction and the recognized emotion data and transmits them to the server.

[1927] User processing steps

[1928] Step 1:

[1929] The user connects to the call center by phone or using an application.

[1930] Step 2:

[1931] After connecting, check the list of waiting operators and select the character you want.

[1932] Step 3:

[1933] The selected animated character will synchronize with the audio in real time and begin interacting.

[1934] Step 4:

[1935] Interact with an operator whose facial expressions change in real time thanks to an emotion engine.

[1936] Step 5:

[1937] After the call is over, the emotional data recognized by the emotion engine is integrated into a feedback form to provide satisfaction and opinions.

[1938] Through the above steps, the system can provide users with an emotional and face-to-face interaction, and the recording of interaction and emotional data provides valuable data for improving service quality.

[1939] Example 2

[1940] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1941] Conventional call center systems mainly rely on voice communication between operators and users, limiting interaction. Furthermore, they are unable to recognize users' emotions in real time and respond accordingly, resulting in a poor user experience. Furthermore, processes such as operator selection and interaction recording are inefficient, making it difficult to improve service quality. There is a need to solve these problems and provide responses that are more emotional and closer to face-to-face interactions.

[1942] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1943] In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the operator's voice in real time, means for transmitting the synchronized voice and character to a user terminal, means for the user to select an operator, means for recognizing the user's emotion, and means for changing the facial expressions and movements of the animated character based on the recognized emotion. This allows the user to receive emotional responses in real time, allowing for an interaction experience that is closer to face-to-face interaction. Furthermore, feedback can be collected based on interaction records and emotion data, making it possible to continuously improve the quality of service.

[1944] "Means for acquiring the operator's voice" refers to devices or software for capturing the voice data spoken by the operator in real time and inputting it into the system.

[1945] "Means for generating animated characters" means technologies and algorithms for generating animated characters that can move in real time based on operator voice and profile data.

[1946] "Means for synchronizing voice and characters in real time" refers to software or systems that allow for timely adjustment and reflection of the movements and expressions of animated characters based on the operator's voice data.

[1947] "Means for transmitting to the user terminal" refers to the communication technology and protocol for transmitting synchronized audio data and animation data from the server to the client terminal in real time.

[1948] "Means for the user to select an operator" refers to a GUI (graphical user interface) on the interface used by the user to select a desired operator from a plurality of operators.

[1949] "Means for recognizing user emotions" refers to an emotion recognition engine or algorithm that processes the user's voice data and facial expression data and analyzes their emotional state.

[1950] "Means for changing the facial expressions and movements of an animated character based on emotions" refers to a control system or algorithm that uses recognized user emotional data to change the facial expressions and movements of an animated character in real time.

[1951] "Means for storing operator information in a database" refers to the functions and systems for registering and storing basic information about operators (such as names, profile photos, and voice samples) in a database.

[1952] "Means for sending a list of waiting operators to a client" refers to a communication means or protocol for generating a list of waiting operators on the server side and sending it to the user's terminal.

[1953] "Means for recording interactions" refers to technologies and systems that record all voice interactions and actions between users and operators for later analysis and evaluation.

[1954] "Means for collecting feedback based on emotional data" refers to a method for analyzing emotional data collected during interactions with users and using the results to collect user satisfaction and areas for improvement through a feedback form, etc.

[1955] "Measures to improve service quality" refers to continuous improvement methods that analyze collected feedback data and use it to improve the way operators respond and the performance of the entire system.

[1956] MODE FOR CARRYING OUT THE INVENTION

[1957] This invention relates to a system that synchronizes voice and animation in real time and recognizes a user's emotions. This system is composed of a server, a terminal (client), and a user.

[1958] Basic Server Configuration

[1959] The server is the central component responsible for the main processing and implements the following functions:

[1960] 1. Register operator information

[1961] The server stores various information about the operator (name, profile photo, voice sample) in a database. An RDBMS such as MySQL is used here. Using Python's OpenAI API library, the voice sample is sent to a generative AI model (e.g., DALL-E) to generate an animated character. The generated character is then stored back in the database.

[1962] 2. Waiting for client connections

[1963] The server waits for and processes connection requests from clients using a communications platform such as Twilio.

[1964] 3. Operator selection and synchronization

[1965] Based on the client's request, the server sends a list of available operators to the client. When the user selects an operator, the server receives that information and synchronizes the operator's voice data with the animated character in real time. APIs such as Amazon Polly and Google Text-to-Speech are used for voice processing.

[1966] 4. Emotion Recognition by Emotion Engine

[1967] The server uses Azure Cognitive Services' Emotion API to recognize emotions from the user's voice and facial expressions, and the recognized emotional data is reflected in the facial expressions and movements of the animated character in real time.

[1968] 5. Voice and character transmission

[1969] The server sends synchronized audio and animated characters to the client using real-time communication such as WebSocket.

[1970] 6. Recording interaction and emotion data

[1971] The server records user interactions and emotional data and stores it in Amazon S3 or Google Cloud Storage, which helps gather subsequent feedback and improve the service.

[1972] Basic configuration of the terminal (client)

[1973] The client terminal is a device used by a user and has the following functions:

[1974] 1. Sending a connection request

[1975] Users use a dedicated smartphone app or web application to send a connection request to a server, which is established through a service such as Twilio.

[1976] 2. Operator Selection

[1977] The user checks the list of waiting operators sent from the server and selects the desired operator, and the selection information is sent to the server.

[1978] 3. Real-time synchronization

[1979] The client receives and displays the selected operator's voice and animated character in real time. The animated character moves in real time using the Unity engine or similar.

[1980] 4. Emotion-aware feedback

[1981] The client receives the user's emotion data recognized by the emotion engine and displays a response based on that data (such as changing the character's facial expression).

[1982] 5. Recording interactions

[1983] The client records the user's interaction and emotion data and sends it to the server, which uses the data to improve satisfaction and services.

[1984] Basic user actions

[1985] A user uses the system in the following steps:

[1986] 1. Connect to the call center

[1987] Users connect to the call center using a phone or an application, with the connection provided through a communications platform such as Twilio.

[1988] 2. Confirm your operator selection

[1989] The user checks the list of waiting operators presented by the server and selects a desired operator.

[1990] 3. Start of the conversation

[1991] The selected animated character will be synchronized with the voice in real time, allowing the user to receive a face-to-face response, and the emotion engine will recognize the user's emotions in real time, providing a more emotionally relevant response.

[1992] 4. Providing emotional feedback

[1993] After the call is completed, a feedback form is displayed based on the emotional data recognized by the emotion engine, allowing the user to provide their satisfaction and opinions.

[1994] Specific examples

[1995] Prompt Sentence Examples

[1996] Example: Use the voice sample of Operator A to generate an animated character using a generative AI model. Voice sample URL: http: / / example.com / audio_sample

[1997] Compared to traditional call centers, the system offers a more empathetic, face-to-face experience, and by recording interactions and collecting feedback, it can continuously improve service quality.

[1998] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1999] Program processing flow

[2000] server

[2001] Step 1: Register operator information

[2002] Input: Operator name, profile picture, voice sample

[2003] What it does: The server receives the operator's name, profile picture, and voice sample from the form and stores the data in a MySQL database.

[2004] Data processing: The server uses Python's OpenAI API library to send voice samples to DALL-E with prompt sentences to generate a corresponding animated character.

[2005] Output: The generated animated characters are saved back to the database.

[2006] Step 2: Waiting for client connections

[2007] Input: Connection request from the client

[2008] Specific operation: The server waits for a call center connection request from a client using a cloud communication platform such as Twilio.

[2009] Data Processing: Manage and update the list of available operators in real time.

[2010] Output: Generates the operator list upon receiving a request from the client.

[2011] Step 3: Select and sync operators

[2012] Input: Operator selection information from the user

[2013] Specific operation: Based on a request from a client, the server sends a list of waiting operators to the client.

[2014] Data processing: The voice of the selected operator is synchronized with the movement data of the corresponding animated character in real time. Amazon Polly is used for voice processing.

[2015] Output: Synchronized data is generated.

[2016] Step 4: Emotion Recognition with the Emotion Engine

[2017] Input: User's voice and facial expressions

[2018] Specific operation: The server uses the Emotion API of Azure Cognitive Services to recognize emotions from the user's voice and facial expressions.

[2019] Data processing: Collect recognized emotion data and update the facial expressions and movements of the animated character.

[2020] Output: Updated animated character data is generated.

[2021] Step 5: Submit your voice and character

[2022] Input: Synchronized audio and animated character data

[2023] Specific operation: The server transmits synchronized audio and animated characters to the client in real time using WebSocket.

[2024] Data processing: Convert the data into a WebSocket communication format.

[2025] Output: Generates data sent to the client.

[2026] Step 6: Recording interaction and emotion data

[2027] Input: User interaction data and recognized emotion data

[2028] Specific Operation: The server records the user's interactions and the recognized emotion data.

[2029] Data processing: The recorded data is stored on Amazon S3 or Google Cloud Storage.

[2030] Output: The saved data is generated.

[2031] Terminal (client)

[2032] Step 1: Sending a Connection Request

[2033] Input: A user requesting a connection

[2034] How it works: A user sends a connection request to a server using a dedicated smartphone app or web application. This connection is made through a service such as Twilio.

[2035] Data processing: Generate connection request data.

[2036] Output: Generates the connection request data sent to the server.

[2037] Step 2: Select an operator

[2038] Input: List of waiting operators from the server

[2039] Specific operation: The user checks the list of waiting operators sent from the server and selects the desired operator. The selection information is sent to the server.

[2040] Data processing: Generate user selection information.

[2041] Output: Selection information is generated that is sent to the server.

[2042] Step 3: Real-time sync

[2043] Input: Synchronized audio and animated character data sent from the server

[2044] Specific operation: The client receives and displays the selected operator's voice and animated character in real time. The animated character moves in real time using the Unity engine or similar.

[2045] Data processing: Converting received data into a playable format.

[2046] Output: Synchronized voice and character data is generated that is displayed to the user.

[2047] Step 4: Emotion recognition feedback

[2048] Input: Emotion data from the server

[2049] Specific operation: The client receives the user emotion data recognized by the emotion recognition engine and displays a response based on that data (such as changing the character's facial expression).

[2050] Data processing: Generate animation update data based on emotion data.

[2051] Output: Updated animated character data is generated that is displayed to the user.

[2052] Step 5: Record the interaction

[2053] Input: User interaction data and emotion data

[2054] Specific operation: After the interaction, the client records the user's interaction data and emotion data and sends them to the server.

[2055] Data processing: Generate recorded interaction data and emotion data.

[2056] Output: Interaction data and emotion data are generated and sent to the server.

[2057] User

[2058] Step 1: Connect to a call center

[2059] Input: A user requesting a connection

[2060] What it does: A user connects to a call center using a phone or a dedicated app, and the connection is made through a communications platform such as Twilio.

[2061] Data processing: Generate connection request data and send it to the server.

[2062] Output: The connection request data sent to the server is generated.

[2063] Step 2: Confirm operator selection

[2064] Input: List of waiting operators from the server

[2065] Specific operation: The user checks the list of waiting operators presented by the server and selects the desired operator.

[2066] Data processing: Generate user selection information and send it to the server.

[2067] Output: Selection information is generated that is sent to the server.

[2068] Step 3: Start the conversation

[2069] Input: Synchronized audio and animated character data sent from the server

[2070] Specific operation: An animated character selected by the user is synchronized with the audio in real time, and the user is treated as if they were face-to-face.

[2071] Data processing: Converting received audio and animation data into a playable format.

[2072] Output: Synchronized voice and character data is generated that is displayed to the user.

[2073] Step 4: Provide emotional feedback

[2074] Input: Emotion data from the server

[2075] Specific operation: Based on the emotion data recognized by the emotion engine, a feedback form is displayed on the user's device. The user provides their satisfaction and opinions.

[2076] Data processing: Generate user feedback data and send it to the server.

[2077] Output: Feedback data is generated that is sent to the server.

[2078] (Application example 2)

[2079] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2080] Conventional customer service systems have the problem that the operator's facial expressions and emotions are difficult to convey, making communication with the user feel mechanical and cold. Furthermore, the lack of a system that can flexibly respond to the user's choices and emotions makes it difficult to provide highly satisfying service. This is particularly true in brick-and-mortar stores, where it is difficult for operators in remote locations to provide effective customer service in real time.

[2081] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the voice of an operator, means for generating an animated character based on the voice, means for synchronizing the generated animated character and the voice of the operator in real time, means for transmitting the synchronized voice and character to a user terminal, means for the user to select an operator, means for recognizing the user's emotion, and means for changing the facial expression of the animated character based on the recognized emotion. This allows the user to enjoy an emotional response similar to that of a face-to-face interaction, thereby enabling highly satisfying customer service.

[2082] "Means for acquiring the operator's voice" means a device or software for receiving the voice signal emitted by the operator and recording and processing it as digital data.

[2083] "Means for generating animated characters" means algorithms or software for automatically creating 2D or 3D animated characters based on operator voice or other data.

[2084] "Means of real-time synchronization" refers to technology or systems that instantly match the operator's voice and the movements of the animated character, displaying them without any time lag.

[2085] The "means for transmitting to the user terminal" is a communication function for transmitting the generated animated character and the audio synchronized therewith to the user's device via a network.

[2086] "Means for a user to select an operator" refers to an interface or system that allows a user to select a specific operator from among multiple operators.

[2087] "Means for recognizing user emotions" refers to algorithms and implementations for analyzing and identifying the emotional state of a user from their facial expressions and voice.

[2088] "Means for changing the facial expression of an animated character" refers to a function or system for appropriately changing the facial expression or movement of an animated character in accordance with the recognized emotions of the user.

[2089] "Means for storing operator information in a database" refers to a database system that centrally manages and stores basic information about operators, voice samples, generated character information, etc.

[2090] The "means for transmitting a list of waiting operators to a client" refers to a function or technology for transmitting a list of currently available operators to a user's device.

[2091] The "means for generating data for simulating information about a product" is a simulation function for automatically generating data required for product explanations and demonstrations.

[2092] The "means for providing product information to users" refers to a display function or interface for effectively conveying the generated product data to users.

[2093] "Means for recording user interactions" refers to a system for saving the content of conversations and actions with users as digital data.

[2094] "Means of collecting feedback" are surveys and rating systems for collecting user satisfaction and opinions.

[2095] "Means for dynamically changing customer service content based on user interaction" refers to algorithms and technologies that adapt and change customer service content in real time according to the user's dialogue and behavior.

[2096] The present invention relates to a system for recognizing a user's emotions by synchronizing an operator's voice with an animated character in real time. A specific embodiment of this system will now be described.

[2097] server

[2098] The server first captures the operator's voice. Commonly used hardware includes a microphone to record the voice as digital data and a server computer to process it. The voice data is captured using digital signal processing (DSP) technology.

[2099] The server then generates an animated character based on the voice data. The software used is a generative AI model (e.g., OpenAI's DALL-E or MidJourney). It analyzes the voice sample and generates an animated character based on its characteristics. This generated character is then stored in a database.

[2100] The server analyzes the voice data and links it with the character's movement data to synchronize the generated animated character and the operator's voice in real time. This synchronization is achieved using real-time voice analysis software and motion capture technology.

[2101] The server has a communication function to transmit synchronized audio and animated characters to the user terminal, using the commonly used Internet Protocol (TCP / IP) as the communication protocol.

[2102] The server also recognizes the user's emotions and dynamically changes the facial expression of the animated character based on that data. This emotion recognition uses the Hugging Face transformers library and DeepFace. The server analyzes emotions from the user's voice and facial expressions, dynamically changing the character's facial expression accordingly.

[2103] In addition, the server stores operator information in a database and has the function of sending a list of waiting operators to the user's terminal. It also plays a role in improving the efficiency of customer service by generating data to simulate product information and providing it to the user.

[2104] Terminal (client)

[2105] The user terminals include smartphones and smart glasses, and connect to the call center using these terminals. The user can send a connection request through the application and check the list of waiting operators. This list is provided by the server, and the user can select the operator they want.

[2106] Once the selection is complete, the voice and animated character are synchronized from the server and displayed in real time on the user's device. The device is equipped with a camera and microphone to recognize the user's emotions. This emotional data is sent to the server, and the character's facial expression changes dynamically based on the content of the conversation.

[2107] User

[2108] Users interact with the virtual store clerk through the terminal, and the system also has the ability to record user interactions and collect feedback, which is used to improve subsequent services.

[2109] Specifically, when a user visits a store and launches the app on their smartphone, a list of available virtual clerks is displayed. The user selects a virtual clerk from the list and begins a conversation. The system then recognizes the user's emotions and changes the virtual clerk's response accordingly. The following is an example of a prompt sentence:

[2110] Example prompt sentence:

[2111] "The user launches the app and gets a list of available virtual sales associates. The user selects a virtual sales associate and begins interacting with them."

[2112] In this way, users can enjoy a more emotional and face-to-face interaction, making it possible to provide a high level of satisfaction that was difficult to achieve with conventional customer service systems.

[2113] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2114] Step 1:

[2115] The server captures the operator's voice. Specifically, when the operator speaks into the microphone, the voice is sent to the server as a digital signal. This voice data is analyzed using digital signal processing (DSP) and saved in a database in real time. The input is the operator's voice through the microphone, and the output is the analyzed digital voice data.

[2116] Step 2:

[2117] The server generates an animated character based on the operator's voice data. It uses a generative AI model (e.g., DALL-E or MidJourney) to generate an image of the animated character from the voice features. This generated character is stored in a database. The input is the operator's voice sample, and the output is an image of the generated animated character.

[2118] Step 3:

[2119] The server synchronizes the generated animated character with the operator's voice in real time. To do this, it uses real-time voice analysis software to analyze the voice data and match the character's mouth movements and facial expressions. The input is digital voice data and an image of the animated character, and the output is a synchronized set of animated character and voice.

[2120] Step 4:

[2121] The server transmits synchronized audio and animated characters to the user device. The communication protocol is HTTP or WebSocket, and data is streamed to the user device in real time. The input is synchronized animated character and audio data, and the output is real-time animation displayed on the user device.

[2122] Step 5:

[2123] The terminal sends a connection request from the user to the server. Specifically, when the user launches an application on their smartphone or smart glasses and presses the connect button, the request is sent to the server. The input is the user's connection request, and the output is the connection request data sent to the server.

[2124] Step 6:

[2125] The server sends a list of waiting operators to the user terminal. It retrieves operator information from the database, generates a list of available operators, and sends it to the user terminal. The input is the operator information stored in the database, and the output is the list of waiting operators displayed on the user terminal.

[2126] Step 7:

[2127] The user selects an operator and sends the selected information to the server through the terminal. The input is the user's selection operation, and the output is the selected information sent to the server.

[2128] Step 8:

[2129] The server analyzes the user's facial expression data received from the device to recognize the user's emotions. DeepFace is used for emotion recognition, and the analysis results are reflected in the character's facial expressions. The input is the user's facial expression data, and the output is the recognized emotional data and the character's facial expressions that change based on it.

[2130] Step 9:

[2131] The server records interactions and collects feedback. The content of the dialogue with the user is saved as digital data and later analyzed. This data is analyzed by the feedback collection system and used to improve the service. The input is the dialogue data with the user, and the output is the analysis results as feedback.

[2132] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2133] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2134] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2135] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2136] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2137] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2138] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2139] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2140] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2141] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2142] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2143] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2144] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2145] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2146] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network ...

Claims

1. a means for acquiring the operator's voice; means for generating an animated character based on the audio; A means of synchronizing the generated animated characters with the operator's voice in real time; means for transmitting the synchronized voice and character to a user terminal; means for a user to select an operator; A system including:

2. a means for storing operator information in a database; means for transmitting a list of waiting operators to the client based on the information; The system of claim 1 further comprising:

3. a means for recording user interactions; The system of claim 1 , further comprising means for collecting feedback based on the recorded interactions.

Images