system

Abstract

We provide the system. [Solution] A means for converting the user's voice into text using a speech recognition engine, A means of analyzing text data using natural language processing technology to understand user intent, A server device that generates a response based on intent and provides the generated response as text data, A terminal device that converts text data into speech and plays it back to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional customer service training, since actual customers or crew members acting as customers are required, there is a problem that it is difficult to provide a flexible and effective training environment. In addition, since the training opportunities with different personas are limited, it is difficult to sufficiently acquire skills to handle various scenarios. Also, in the current training method, feedback and advice are insufficient, and skill improvement often stagnates. Furthermore, since there is no means to quantitatively evaluate the progress, there is also a problem that it is difficult to measure the effect of training. To solve these problems, the present invention develops a system that provides a flexible and effective customer service training environment and supports the skill improvement of crew members.

Means for Solving the Problems

[0005] The present invention solves the above problems by the following means. The system of the present invention includes means for converting the user's voice into text using a speech recognition engine, means for analyzing the text data using natural language processing technology to understand the user's intent, a server device that generates a response based on the intent and provides the generated response as text data, and a terminal device that converts the text data into speech and plays it back to the user. It also includes a terminal device that provides an interface for selecting persona information, a server device that prepares dialogue patterns based on the selected persona, and a server device that starts a scenario based on the dialogue pattern and generates a voice response according to the scenario. Furthermore, it includes means for recording and evaluating the user's utterances, a server device that calculates points based on the evaluation and updates the points, and a terminal device that displays points and performance. As a result, crew members can hone their skills in an environment close to actual customer service scenes and practice effectively.

[0006] A "speech recognition engine" refers to software or hardware used to analyze speech data and convert it into text data.

[0007] "User" refers to a person who uses this system to conduct customer service training.

[0008] "Text data" refers to string information converted from audio data by a speech recognition engine.

[0009] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human language.

[0010] A "response generation algorithm" refers to the computational procedures and logic used to generate an appropriate response based on the analyzed user intent.

[0011] A "server device" refers to a computer system that provides functions such as data processing, storage, and network communication.

[0012] A "terminal device" refers to an electronic device that allows a user to access a system using an interface.

[0013] A "speech synthesis engine" refers to software or hardware used to convert text data into speech data.

[0014] "Persona information" refers to character setting information based on specific customer types and situations.

[0015] A "dialogue pattern" refers to a pre-set combination of questions and answers based on a persona or scenario.

[0016] A "scenario" refers to training content that progresses through interaction with the user, based on a pre-set storyline.

[0017] "Evaluation criteria" refer to specific indicators and rules used to evaluate user utterances, response times, and other factors.

[0018] "Points" refer to numerical rewards or performance indicators given based on user evaluations. [Brief explanation of the drawing]

[0019] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0020] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0021] First, the language used in the following description will be explained.

[0022] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0023] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0024] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0025] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0027] [First Embodiment]

[0028] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0029] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0032] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0035] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0039] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0040] This invention relates to a customer service training system that utilizes voice dialogue AI technology. This system aims to improve users' customer service skills through natural conversation using speech recognition and natural language processing (NLP). Specific embodiments of this invention are described below.

[0041] System Configuration

[0042] 1. Server equipment

[0043] Speech recognition engine

[0044] Natural Language Processing Engine

[0045] Response generation algorithm

[0046] Evaluation and point calculation system

[0047] Database (user data, persona information, dialogue patterns)

[0048] 2. Terminal device

[0049] Microphone and speaker

[0050] Speech synthesis engine

[0051] Point and feedback display interface

[0052] Persona Selection Interface

[0053] Specific steps for implementing the invention

[0054] Users of this system use a terminal device to begin customer service training. When a user starts a customer service scenario, the following processes occur.

[0055] 1. Voice recording and recognition

[0056] The (device) displays a recording start button, and the (user) presses the recording start button.

[0057] The (device) uses the microphone to record the (user's) voice.

[0058] The (device) sends the recorded audio data to the (server).

[0059] The server uses a speech recognition engine to convert the speech data into text data.

[0060] 2. Natural Language Processing and Response Generation

[0061] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[0062] The server generates an appropriate response based on the intent and sends that response as text data to the terminal.

[0063] The (device) converts text data into speech using a speech synthesis engine and plays the response back to the (user).

[0064] 3. Persona-based training

[0065] The user uses the terminal device's persona selection interface to select a specific customer type or persona (e.g., "Customer with a complaint").

[0066] The (device) sends the selected persona information to the (server).

[0067] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[0068] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[0069] The (device) converts the text data into speech and plays it back to the user.

[0070] As the training progresses, various persona-based dialogue scenarios unfold seamlessly.

[0071] 4. Evaluation and Feedback

[0072] The (terminal) records the user's speech in real time and sends it to the (server).

[0073] The server scores the user's utterances based on evaluation criteria.

[0074] The server calculates points based on the evaluation results and updates the user's performance database.

[0075] The server generates a feedback message and sends it to the terminal.

[0076] The (device) displays and provides a feedback message to the user.

[0077] Specific example

[0078] Example 1: Product Description Practice

[0079] The user selects a persona for the "product description" and starts the scenario.

[0080] The (device) started recording, and the AI ​​bot said, "Could you please explain this new product?"

[0081] The user says, "This product uses the latest technology..."

[0082] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[0083] The (device) plays the question aloud and continues the conversation.

[0084] Example 2: Practicing handling complaints

[0085] The user selects the "Complaint Handling" persona and starts the scenario.

[0086] The (device) started recording, and the AI ​​bot said, "This product arrived damaged, so could I request an exchange?"

[0087] The user says, "I am very sorry, I will take care of it immediately."

[0088] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[0089] The (device) plays the question aloud and continues the conversation.

[0090] Thus, this system is designed to allow users to efficiently acquire the skills necessary to handle a wide range of scenarios while actually engaging in dialogue.

[0091] The following describes the processing flow.

[0092] Utilization of voice-activated AI bots

[0093] Step 1:

[0094] The user presses the recording start button on the terminal device.

[0095] The (device) uses the microphone to record the user's voice.

[0096] Step 2:

[0097] The (device) saves the recorded audio data to a buffer.

[0098] The (terminal) sends the audio data in the buffer to the (server).

[0099] Step 3:

[0100] The server starts the speech recognition engine and converts the received speech data into text data.

[0101] The server generates the text data of the conversion result.

[0102] Step 4:

[0103] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[0104] The server generates a response based on the intended meaning.

[0105] Step 5:

[0106] The server generates a response as text data and sends it to the terminal.

[0107] The text data received by the terminal is converted into speech using a speech synthesis engine.

[0108] The (device) plays the converted audio through the speaker.

[0109] Persona-based training

[0110] Step 1:

[0111] The user uses the terminal device's interface to select a specific customer type or persona.

[0112] The (device) sends the selected persona information to the (server).

[0113] Step 2:

[0114] The server processes the persona information it receives.

[0115] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[0116] Step 3:

[0117] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[0118] The (device) receives text data, converts it into speech, and plays it back to the (user).

[0119] Role-playing progress evaluation and point system

[0120] Step 1:

[0121] The (device) records the user's speech in real time.

[0122] The (terminal) sends the recorded speech data to the (server).

[0123] Step 2:

[0124] The server analyzes the received speech data and evaluates its appropriateness and timing based on pre-set evaluation criteria.

[0125] The server calculates a score for each evaluation item.

[0126] Step 3:

[0127] The server calculates points based on the evaluation results.

[0128] The server adds points and updates the user's performance data.

[0129] Step 4:

[0130] The server sends the generated points and performance information to the terminal.

[0131] The terminal displays the points and performance information it has received and provides it to the user.

[0132] Feedback and improvement loop

[0133] Step 1:

[0134] The server re-evaluates the user's utterances and generates specific feedback.

[0135] The server sends the generated feedback message to the terminal as text data.

[0136] Step 2:

[0137] The (device) displays the feedback message it received on the screen.

[0138] The user reviews the feedback message and understands the areas for improvement.

[0139] (Example 1)

[0140] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0141] In customer service training, it is crucial for users to receive practical training based on diverse scenarios in order to effectively improve their conversational skills. However, conventional systems have not effectively performed real-time speech recognition or natural language processing, making it difficult to accurately understand and respond to user intentions. Furthermore, methods for evaluating user speech and providing feedback were insufficient, making it difficult for users to objectively evaluate their own skills and continuously improve them. Therefore, a new system is needed to effectively improve users' customer service skills.

[0142] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0143] In this invention, the server includes speech recognition means for converting user speech into text data, natural language processing means for analyzing the text data and understanding the user's intent, and response generation means for generating a response based on the user's intent and providing the generated response as text data. This enables users to engage in real-time dialogue based on diverse scenarios and effectively improve their customer service skills. Furthermore, by including means for providing an interface for selecting persona information, means for preparing dialogue patterns based on the selected persona, and means for starting a scenario based on the dialogue pattern and generating a voice response corresponding to the scenario, users can receive practical training according to diverse scenarios. In addition, by including means for recording and evaluating the user's utterances, means for calculating and updating points based on the evaluation, and means for displaying points and performance, users can objectively evaluate their own skills and continuously improve them.

[0144] "Voice recognition means" refers to a device or technology that records a user's voice and converts it into text data.

[0145] "Natural language processing" refers to technologies that analyze text data and understand the user's intent.

[0146] "Response generation means" refers to a device or technology that generates an appropriate response based on the user's intent and provides the generated response as text data.

[0147] "Speech synthesis means" refers to a device or technology that converts text data into speech and plays it back to the user.

[0148] "Persona information" refers to character information selected by specific customer types or users.

[0149] A "dialogue pattern" is a set of dialogue sequences and response patterns prepared based on a specific scenario.

[0150] A "scenario" is a sequence of dialogues based on a specific situation or case.

[0151] "Means for recording user speech content" refers to a device or technology that records the voice spoken by a user and stores it as data.

[0152] "Evaluation criteria" refer to the standards and rules used to evaluate the content of a user's utterances, and scoring is performed based on these criteria.

[0153] "Means for calculating points" refers to a device or technology that quantifies the user's utterances based on evaluation criteria and calculates them as points.

[0154] "Means for updating points" refers to a device or technology that reflects the calculated points in the user's performance database and updates it to the latest state.

[0155] An "interface" is a user interface or input device that allows a user to interact with a system.

[0156] A "feedback tool" is a device or technology that provides users with evaluation results, areas for improvement, and other information to help them improve their own skills.

[0157] This invention is a customer service training system that utilizes a speech recognition engine and natural language processing (NLP) technology. The system includes three main components: a server, a terminal, and a user.

[0158] First, the server includes a speech recognition engine, a natural language processing engine, a response generation algorithm, an evaluation and point calculation system, and a database that stores user data, persona information, and dialogue patterns. For the speech recognition engine, for example, Google® Cloud Speech-to-Text API can be used. For natural language processing, technologies such as spaCy or BERT are recommended. The response generation algorithm generates appropriate responses based on the user's intent and provides them as text data. The server also performs evaluations based on the user's utterances, calculates points, and generates feedback.

[0159] Next, the device includes a microphone and speaker, a speech synthesis engine, a point and feedback display interface, and a persona selection interface. For the speech synthesis engine, for example, Amazon Polly or Google Text-to-Speech can be used. The device records the user's voice and sends it to the server in real time. The text data received from the server is converted into speech and played back to the user. Furthermore, the persona selection interface allows the user to select specific customer types or personas.

[0160] The user receives customer service training using a device. For example, if the user selects the "product description" persona, the device starts recording and asks a voice question such as, "Could you please explain this new product?" When the user says, "This product uses the latest technology...", the device records the voice and sends it to the server. The server uses a speech recognition engine to convert the voice to text, analyzes it with a natural language processing engine, and then generates an appropriate response, "Could you tell me how to use it specifically?", and sends it to the device. The device converts that text data back into voice and plays it back to the user.

[0161] Furthermore, user performance is evaluated and calculated as points. The evaluation criteria are based on factors such as the content, timing, and politeness of the speech. Along with the evaluation results, points and feedback are provided to the user. For example, feedback such as "Your response this time was excellent" might be displayed.

[0162] Example of a prompt:

[0163] "Select a persona for your product description and begin explaining the new product."

[0164] "Let's select a persona for handling complaints and ask them to exchange the product."

[0165] In this way, users can efficiently improve their practical customer service skills through real-time interactions.

[0166] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0167] Step 1: Record and recognize audio

[0168] Input: User's speech

[0169] Operation: The device displays a recording start button, and recording begins when the user presses it. For example, the user might say, "Please explain the new product."

[0170] Data processing: The device's microphone records the user's voice and generates audio data.

[0171] Output: Recorded audio data

[0172] Operation: The device sends the recorded data to the server, and the server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio data into text data.

[0173] Step 2: Natural Language Processing and Response Generation

[0174] Input: Text data (converted from user's speech)

[0175] Operation: The server uses a natural language processing (NLP) engine (e.g., spaCy or BERT) to analyze text data and understand the user's intent. For example, it might analyze a request like "Please explain the new product."

[0176] Data processing: The NLP engine analyzes text data to identify the user's intent.

[0177] Output: User intent (e.g., the user is seeking a product description)

[0178] Operation: The server's response generation algorithm generates an appropriate response based on the user's intent and sends the generated response as text data to the terminal. For example, a response such as "What are its features?" might be generated.

[0179] Step 3: Speech synthesis and response playback

[0180] Input: Text data of the response

[0181] Operation: The device converts the received text data into speech using a text-to-speech engine (e.g., Amazon Polly or Google Text-to-Speech).

[0182] Data processing: The speech synthesis engine converts text data into speech data.

[0183] Output: Audio data

[0184] Operation: The device plays audio data and communicates a response to the user. For example, it might play the audio "What are its features?".

[0185] Step 4: Persona-based training

[0186] Input: Persona information selected by the user

[0187] Operation: The user selects a specific customer type or persona using the device's persona selection interface. For example, they might select "Customers with complaints."

[0188] Data processing: The device sends the selected persona information to the server, and the server prepares dialogue patterns based on the persona information.

[0189] Output: Dialogue patterns based on the selected persona

[0190] Operation: The server generates a scenario based on the dialogue pattern and sends the initial question and situation description as text data to the terminal. The terminal converts the text data into speech and plays the scenario. For example, it plays a scenario such as, "This product arrived damaged, could I request a replacement?"

[0191] Step 5: Record and evaluate the content of the speech.

[0192] Input: User's utterance

[0193] Operation: The device records the user's speech in real time and sends it to the server.

[0194] Data processing: The server scores the user's speech based on evaluation criteria. For example, it evaluates elements such as the content of the speech, timing, and politeness.

[0195] Output: Evaluation results and points

[0196] Operation: The server calculates points based on the evaluation results and updates the points.

[0197] Step 6: Generating and providing feedback

[0198] Input: Evaluation results and points

[0199] Operation: The server generates a feedback message and sends it to the terminal to inform the user.

[0200] Data processing: The feedback message is generated as text data.

[0201] Output: Feedback message

[0202] Action: The device displays and provides a feedback message to the user. For example, the feedback might say, "The service you provided was excellent."

[0203] In this way, the system effectively trains users' customer service skills through a series of steps, and provides evaluation and feedback.

[0204] (Application Example 1)

[0205] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0206] Conventional customer service training systems use speech recognition and natural language processing technologies to interact with users, but they have the drawback of not being able to evaluate actions and speech during training in real time. Furthermore, they lacked training that took user actions into account and detailed feedback on speech content. As a result, there was a problem in that improvements in customer service skills in actual stores were not sufficiently achieved.

[0207] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0208] In this invention, the server includes means for converting the user's speech into text using a speech recognition engine; means for analyzing the text data using natural language processing technology to understand the user's intent; means for generating a response based on the intent and providing the generated response as text data; means for capturing the user's actions with a camera and evaluating the synchronized speech content and actions; means for generating a response based on the user's speech content using a generative AI model; means for providing the generated response to the user as evaluation and feedback; and means for storing the speech content and evaluation results in a database. This makes it possible to grasp the user's actions and speech content in real time and provide evaluation and feedback on the spot.

[0209] A "speech recognition engine" is a device or software that captures a user's speech as a digital signal and converts that speech into text data.

[0210] "Natural language processing technology" is a technology that analyzes text data, understands the user's intentions and emotions, and enables natural dialogue between humans and computers.

[0211] "Means for generating responses" refer to algorithms or systems that construct appropriate responses based on the user's intent and provide them as text or audio data.

[0212] A "server device" is a central processing unit that receives and processes data from multiple users and provides necessary information and services.

[0213] A "terminal device" is a device that a user directly operates, equipped with a microphone, speaker, display, etc., and functions in conjunction with a server.

[0214] A "camera" is a device that captures a user's actions and gestures in real time and transmits that information to a server as image data.

[0215] A "generative AI model" is an artificial intelligence model that learns from a large dataset and generates natural-sounding responses based on the user's utterances.

[0216] A "prompt sentence" is an input sentence used by a generative AI model to generate a dialogue response, and it is an instruction sentence that appropriately reflects the user's intent.

[0217] A "database" is an information management system that stores and manages information such as spoken content, evaluation results, and dialogue history, and allows for quick reference as needed.

[0218] "Feedback" refers to information provided in real time regarding the evaluation and areas for improvement of the user's speech and actions during training.

[0219] This invention relates to a customer service training system using smart glasses, which provides real-time feedback utilizing speech recognition, natural language processing, and generative AI models. The system consists of a server device, terminal devices, and a network connecting them. Specific embodiments for carrying out the invention are described below.

[0220] System Configuration

[0221] The system consists of the following elements:

[0222] 1. Server equipment:

[0223] Speech recognition engine: Software that converts a user's speech into text data.

[0224] Natural Language Processing (NLP) technology: Software that analyzes text data to understand user intent.

[0225] Generative AI Model: This is an artificial intelligence model that generates responses based on the user's utterances. Specifically, it uses the Hugging Face transformers library and the GPT-3(registered trademark) .5 model.

[0226] Database: A system that stores and manages spoken content, evaluation results, and dialogue history.

[0227] Response generation algorithm: An algorithm that generates an appropriate response based on the results of natural language processing.

[0228] Evaluation and Point Calculation System: A system that evaluates users in real time based on their speech and actions and calculates points.

[0229] 2. Terminal device (smart glasses):

[0230] Camera and microphone: Devices for capturing user actions and voice in real time.

[0231] Text-to-speech engine: Software that converts text data received from a server into speech and plays it back. Specifically, it uses the gTTS library.

[0232] Feedback display interface: An interface for displaying evaluation results and areas for improvement to the user.

[0233] Persona Selection Interface: An interface for users to select training scenarios.

[0234] Operation of the invention

[0235] 1. Customer service training begins:

[0236] The user wears smart glasses and selects a training scenario using a persona selection interface. This sends persona information to the server.

[0237] 2. Speech recognition and motion capture:

[0238] The smart glasses' microphone records the user's voice, and the camera captures their movements. The recorded audio data is sent to a server and converted into text data by a speech recognition engine.

[0239] 3. Natural Language Processing and Response Generation:

[0240] The server analyzes the text data and uses a generative AI model to generate an appropriate response. The generated response is then sent to the terminal device as text data.

[0241] 4. Speech synthesis and playback:

[0242] The terminal device's speech synthesis engine converts text data into speech and plays it back to the user.

[0243] 5. Evaluation and Feedback:

[0244] The server evaluates the user's speech and actions in real time and generates feedback. The evaluation results are stored in a database, and the feedback is provided to the user through a terminal device.

[0245] Specific example

[0246] As a concrete example, let's explain the introduction of a new product. When a user introduces a new product, the system operates in the following order:

[0247] 1. Scenario Selection:

[0248] The user selects "Introduce new products".

[0249] 2. Audio and motion capture:

[0250] The smart glasses capture the user's voice and movements, and the server analyzes them.

[0251] 3. Response generation:

[0252] User: "This product is a convenient product that uses the latest technology."

[0253] Server: The generation AI model generates a response based on the prompt text and responds with "What specific technologies are being used?".

[0254] 4. Audio Playback:

[0255] The smart glasses play the response aloud.

[0256] 5. Evaluation and Feedback:

[0257] The server evaluates the user's speech in real time and provides feedback such as, "It would be good if you could explain the product's features in more detail."

[0258] Example of a prompt

[0259] "The user is describing a new product. Please generate a natural-sounding continuation of the conversation. User: 'This product is a convenient product that uses the latest technology.' Please generate the AI's subsequent response."

[0260] This invention allows staff working in physical stores to efficiently improve their customer service skills while receiving real-time evaluation of user actions and speech. Furthermore, the use of a generative AI model enhances the naturalness and diversity of the dialogue.

[0261] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0262] Step 1:

[0263] The server receives information about the training scenario selected by the user (e.g., "Introduction to a New Product") from the persona selection interface of the terminal device (smart glasses). The input data is the persona information selected by the user, and the server prepares dialogue patterns corresponding to that persona as output.

[0264] Step 2:

[0265] The device uses a microphone built into the smart glasses to record the user's speech in real time. The input data is the voice spoken by the user, and the output data is the digital signal of that voice. This digital signal is sent to a server.

[0266] Step 3:

[0267] The server analyzes the received audio data using a speech recognition engine and converts it into text data. The input data is a digital audio signal, and the output data is the text data obtained by analyzing that audio signal.

[0268] Step 4:

[0269] The server analyzes text data using natural language processing techniques to understand the user's intent. The input data is text data obtained by a speech recognition engine, and the output data is metadata containing the user's intent as a result of the analysis.

[0270] Step 5:

[0271] The server uses a generative AI model based on the user's intent to generate an appropriate response. The input data consists of metadata and prompts for the generative AI model, parsed using natural language processing techniques, while the output data is the text data of the generated response.

[0272] Step 6:

[0273] The server sends the generated response text data to the terminal device. The input data is the response text data, and the output is the transmission of the text data to the terminal device.

[0274] Step 7:

[0275] The terminal converts the received response text data into speech using a speech synthesis engine and plays it back to the user. The input data is text data received from the server, and the output data is speech generated by speech synthesis.

[0276] Step 8:

[0277] The device uses the smart glasses' camera to capture the user's movements and sends the video data to the server. The input data is the video captured by the camera, and the output data is the digital signal of that video.

[0278] Step 9:

[0279] The server evaluates the user's speech and motion data and generates feedback. The input data consists of audio and video data transmitted from the smart glasses, and the output data is a feedback message representing the evaluation result.

[0280] Step 10:

[0281] The server sends the generated feedback message to the terminal device and plays it back to the user either visually or audibly. The input data is the feedback message, and the output data is the transmission of the message to the terminal device and its display or audibility.

[0282] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0283] The present invention is a customer service training system that combines voice dialogue AI technology and emotion recognition. This system not only recognizes the user's voice and understands the user's intention based on natural language processing technology, but also recognizes the user's emotion using an emotion engine. Based on this information, appropriate responses are generated to improve skills. The following shows specific embodiments of the present invention.

[0284] System Configuration

[0285] 1. Server Device

[0286] Speech Recognition Engine

[0287] Natural Language Processing Engine

[0288] Emotion Engine

[0289] Response Generation Algorithm

[0290] Evaluation and Point Calculation System

[0291] Database (user data, persona information, dialogue patterns, emotion data)

[0292] 2. Terminal Device

[0293] Microphone and Speaker

[0294] Speech Synthesis Engine

[0295] Point and Feedback Display Interface

[0296] Persona Selection Interface

[0297] Specific Process for Implementing the Invention

[0298] The user of this system uses the terminal device to start customer service training including emotion recognition. When the user starts a scenario, the following processes are performed.

[0299] 1. Voice recording and recognition

[0300] (The user) presses the recording start button of the terminal device.

[0301] (The terminal) uses the microphone to record the user's voice.

[0302] (The terminal) sends the recorded voice data to the (server).

[0303] (The server) uses a voice recognition engine to convert the voice data into text data.

[0304] 2. Natural language processing and response generation

[0305] (The server) analyzes the text data with a natural language processing (NLP) engine to understand the user's intention.

[0306] (The server) generates an appropriate response based on the intention and sends the response to the (terminal) as text data.

[0307] (The terminal) converts the text data into voice with a text-to-speech engine and plays the response to the user.

[0308] 3. Emotion recognition

[0309] (The server) passes the voice data and text data to an emotion engine to analyze the user's emotion.

[0310] (The server) adjusts the response based on the emotion data to generate a more appropriate conversation.

[0311] 4. Persona-based training

[0312] (The user) selects a persona on the terminal device (e.g., "Customer with a complaint").

[0313] The (device) sends the selected persona information to the (server).

[0314] The server references persona-based dialogue patterns and prepares the scenario.

[0315] The server generates the first question according to the scenario and sends it to the terminal as text data.

[0316] The (device) converts the text data into speech and plays it back to the user.

[0317] 5. Evaluation and Point System

[0318] The (terminal) records the user's speech in real time and sends it to the (server).

[0319] The server scores the speech content and emotional data based on evaluation criteria.

[0320] The server calculates points based on the evaluation results and updates the user's performance database.

[0321] The server generates a feedback message and sends it to the terminal.

[0322] The (device) displays and provides a feedback message to the user.

[0323] Specific example

[0324] Example 1: Product Description Practice

[0325] The user selects a persona for the "product description" and starts the scenario.

[0326] The (device) started recording, and the AI ​​bot said, "Could you please explain this new product?"

[0327] The user says, "This product uses the latest technology..."

[0328] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[0329] The server recognizes the user's emotions from their voice and generates feedback that corresponds to those emotions.

[0330] The (device) plays back questions and feedback via audio, continuing the conversation.

[0331] Example 2: Practicing handling complaints

[0332] The user selects the "Complaint Handling" persona and starts the scenario.

[0333] The (device) started recording, and the AI ​​bot said, "This product arrived damaged, so could I request an exchange?"

[0334] The user says, "I am very sorry, I will take care of it immediately."

[0335] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[0336] The server uses an emotion engine to recognize the user's anger state and generates an additional response to calm them down.

[0337] The (device) plays the question and additional responses aloud, continuing the conversation.

[0338] This system is designed to allow users to efficiently acquire the skills necessary to handle a wide range of scenarios while engaging in actual conversations. Furthermore, the emotion recognition function enables the creation of even more realistic conversation scenarios.

[0339] The following describes the processing flow.

[0340] Utilization of voice-activated AI bots and emotion recognition

[0341] Step 1:

[0342] The user presses the recording start button on the terminal device.

[0343] The (device) uses the microphone to record the user's voice.

[0344] Step 2:

[0345] The (device) saves the recorded audio data to a buffer.

[0346] The (terminal) sends the audio data in the buffer to the (server).

[0347] Step 3:

[0348] The server starts the speech recognition engine and converts the received speech data into text data.

[0349] The server generates the text data of the conversion result.

[0350] Step 4:

[0351] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[0352] The server generates a response based on the intended meaning.

[0353] Step 5:

[0354] The server passes voice and text data to the emotion engine, which analyzes the user's emotions.

[0355] The server adjusts its response based on the emotion recognition results.

[0356] Step 6:

[0357] The server generates a pre-arranged response as text data and sends it to the terminal.

[0358] The text data received by the terminal is converted into speech using a speech synthesis engine.

[0359] The (device) plays the converted audio through the speaker.

[0360] Persona-based training

[0361] Step 1:

[0362] The user uses the terminal device's interface to select a specific customer type or persona.

[0363] The (device) sends the selected persona information to the (server).

[0364] Step 2:

[0365] The server processes the persona information it receives.

[0366] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[0367] Step 3:

[0368] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[0369] The (device) receives text data, converts it into speech, and plays it back to the (user).

[0370] Role-playing progress evaluation and point system

[0371] Step 1:

[0372] The (device) records the user's speech in real time.

[0373] The (terminal) sends the recorded speech data to the (server).

[0374] Step 2:

[0375] The server analyzes the received speech data and evaluates its appropriateness and timing based on pre-set evaluation criteria.

[0376] The server calculates a score for each evaluation item based on speech data and emotion recognition data.

[0377] Step 3:

[0378] The server calculates points based on the evaluation results.

[0379] The server adds points and updates the user's performance data.

[0380] Step 4:

[0381] The server sends the generated points and performance information to the terminal.

[0382] The terminal displays the points and performance information it has received and provides it to the user.

[0383] Feedback and improvement loop

[0384] Step 1:

[0385] The server re-evaluates the user's utterance and generates specific feedback.

[0386] The server sends the generated feedback message to the terminal as text data.

[0387] Step 2:

[0388] The (device) displays the feedback message it received on the screen.

[0389] The user reviews the feedback message and understands the areas for improvement.

[0390] (Example 2)

[0391] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0392] Conventional customer service training systems could understand user intent and generate responses using speech recognition and natural language processing technologies, but they struggled to respond while considering user emotions. Furthermore, there were few means to evaluate the effectiveness of the training, making it difficult to efficiently improve user performance and skills. Additionally, training based on specific scenarios was limited, making it difficult to simulate real-world customer service situations.

[0393] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0394] In this invention, the server includes means for converting user speech to text using a speech recognition engine, means for analyzing text data using natural language processing technology to understand user intent, means for generating a response based on intent and providing the generated response as text data, means for converting text data to speech and playing it back to the user, means for analyzing speech and text data to recognize user emotions, means for adjusting responses based on emotions to generate more appropriate dialogue, means for providing an interface for selecting persona information, means for preparing dialogue patterns based on the selected persona, means for starting a scenario based on the dialogue pattern and generating a speech response corresponding to the scenario, means for recording and evaluating user utterances, means for calculating points based on evaluation and updating points, and means for displaying points and performance. This enables the provision of appropriate responses that take emotions into account, improves the quality of dialogue, and allows for evaluation of user skills and confirmation of performance. Furthermore, by realizing training based on specific scenarios, it becomes possible to simulate situations that closely resemble actual customer service scenarios.

[0395] A "speech recognition engine" is a technology that takes speech data as input and converts it into text data.

[0396] "Natural language processing" is a technology that analyzes text data, understands human language, and generates appropriate responses.

[0397] A "server device" is a computer system that provides data processing and storage functions and operates in conjunction with client devices.

[0398] A "terminal device" is a device that a user directly operates and that provides interface functions while communicating with a server.

[0399] An "emotion engine" is a technology that analyzes voice and text data to estimate a user's emotional state.

[0400] A "persona" is a fictional character based on a specific scenario or user group, and is the subject of simulation in a training system.

[0401] A "dialogue pattern" is a definition of the expected sequence of questions and answers in a particular scenario.

[0402] "Evaluation criteria" are standards or standards used to evaluate user performance based on the content of user utterances and emotional state.

[0403] "Feedback" refers to evaluation comments and advice provided regarding a user's performance.

[0404] System Overview

[0405] This invention relates to a customer service training system that combines voice dialogue AI technology and emotion recognition. This system recognizes the user's voice and understands their intentions based on natural language processing technology, as well as recognizing the user's emotions using an emotion engine. Based on this information, it generates appropriate responses to improve skills.

[0406] Hardware and software configuration

[0407] This system consists of the following components:

[0408] Server equipment

[0409] Speech recognition engine: Converts speech data into text (e.g., Google Cloud Speech-to-Text)

[0410] Natural language processing engine: Analyzes text data and understands user intent (e.g., GPT-3).

[0411] Emotion engine: Analyzes user emotions based on voice and text data (e.g., Microsoft® Azure® Emotion API)

[0412] Response generation algorithm: Generates appropriate responses based on the user's intent and emotions.

[0413] Evaluation and point calculation system: Evaluates the user's utterances and calculates points.

[0414] Database: Stores user data, persona information, dialogue patterns, and sentiment data.

[0415] Terminal device

[0416] Microphone and speaker: Used for audio input and output.

[0417] Speech synthesis engine: Converts text data into speech (e.g., Amazon Polly)

[0418] Point and feedback display interface: Displays points and feedback to the user.

[0419] Persona Selection Interface: An interface for users to select a persona.

[0420] Specific implementations of the system

[0421] Users of this system use a terminal device to begin customer service training, including emotion recognition. Specific embodiments of the present invention are shown below.

[0422] Voice recording and recognition

[0423] The user presses the recording start button on the terminal device.

[0424] The device uses the microphone to record the user's voice.

[0425] The device sends the recorded audio data to the server.

[0426] The server uses a speech recognition engine to convert the audio data into text data.

[0427] Natural language processing and response generation

[0428] The server analyzes the text data using a natural language processing engine to understand the user's intent.

[0429] The server generates an appropriate response based on the user's intent and sends that response as text data to the terminal.

[0430] The device converts text data into speech using a speech synthesis engine and plays the response back to the user.

[0431] emotion recognition

[0432] The server passes voice and text data to the emotion engine, which then analyzes the user's emotions.

[0433] The server adjusts its response based on emotional data to generate more appropriate dialogue.

[0434] Persona-based training

[0435] The user selects a persona (e.g., "a customer with a complaint") on the terminal device.

[0436] The device sends the selected persona information to the server.

[0437] The server references persona-based dialogue patterns and prepares the scenario.

[0438] The server generates an initial question based on the scenario and sends it to the terminal as text data.

[0439] The device converts text data into speech and plays it back to the user.

[0440] Rating and point system

[0441] The device records the user's speech in real time and sends it to the server.

[0442] The server scores the spoken content and emotional data based on evaluation criteria.

[0443] The server calculates points based on the evaluation results and updates the user's performance database.

[0444] The server generates a feedback message and sends it to the terminal.

[0445] The device displays and provides a feedback message to the user.

[0446] Specific usage examples

[0447] Product description practice

[0448] The user selects a persona for the "product description" and starts the scenario.

[0449] The device starts recording and says, "Could you please explain this new product?"

[0450] The user says, "This product uses the latest technology..."

[0451] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[0452] The server recognizes emotions from the user's voice and generates emotionally appropriate feedback.

[0453] The device plays back questions and feedback via voice, continuing the conversation.

[0454] Practice in handling complaints

[0455] The user selects the "complaint handling" persona and starts the scenario.

[0456] The device starts recording and says, "This product arrived damaged, so could I request a replacement?"

[0457] The user says, "I am very sorry, I will take care of it immediately."

[0458] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[0459] The server uses an emotion engine to recognize the user's anger level and generates additional responses to calm them down.

[0460] The device plays back the question and any additional responses via audio, continuing the conversation.

[0461] Examples of prompts for generative AI models

[0462] "Please provide a detailed explanation of the new product."

[0463] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0464] Step 1:

[0465] Audio recording and transfer

[0466] The user presses the recording start button on the terminal device. The input is the user's voice.

[0467] The device uses the microphone to record the user's voice. The output is the recorded audio data.

[0468] The device sends the recorded audio data to the server. The audio data is transferred from the device to the server.

[0469] Step 2:

[0470] Text conversion of audio data

[0471] The server uses a speech recognition engine to convert audio data into text data. The input is recorded audio data.

[0472] A speech recognition engine (such as Google Cloud Speech-to-Text) analyzes the audio data and converts it into text data. The output is text data.

[0473] Step 3:

[0474] Natural language processing and intent analysis

[0475] The server analyzes text data using a natural language processing engine. The input is text data.

[0476] A natural language processing engine (such as GPT-3) analyzes text data to understand the user's intent. The output is data that reflects the user's intent.

[0477] Step 4:

[0478] Response generation

[0479] The server generates an appropriate response based on the user's intent. The input is parsed intent data.

[0480] The response generation algorithm creates a text response tailored to the user's intent. The output is the text data of the response.

[0481] Step 5:

[0482] Speech and playback of responses

[0483] The server sends the generated text data to the terminal. The input is the response text data.

[0484] The device converts text data into speech using a speech synthesis engine (such as Amazon Polly). The output is the generated speech data.

[0485] The terminal plays the generated audio to the user. The audio data is provided to the user through the audio output device.

[0486] Step 6:

[0487] emotion recognition

[0488] The server passes audio and text data to the emotion engine. The input consists of audio and text data.

[0489] An emotion engine (such as the Microsoft Azure Emotion API) analyzes the data and recognizes the user's emotions. The output is emotion data.

[0490] The server adjusts its response based on sentiment data and generates a more appropriate dialogue. Adjusted response data is then generated.

[0491] Step 7:

[0492] Persona-based training begins.

[0493] The user selects a persona (e.g., "Customer with a complaint") on the terminal device. Persona information is then entered.

[0494] The device sends the selected persona information to the server. The input is the selected persona data.

[0495] The server references persona-based dialogue patterns and prepares a scenario. The output is scenario data.

[0496] The server generates the first question based on the scenario and sends it to the terminal as text data. The input is the scenario data.

[0497] The device converts text data into speech and plays it back to the user. The output is audio data.

[0498] Step 8:

[0499] Recording and evaluation of spoken content

[0500] The terminal records the user's speech in real time and sends it to the server. The input is the user's speech.

[0501] The server scores the utterance content and emotional data based on evaluation criteria. The input consists of the utterance content and emotional data.

[0502] The server calculates points based on the evaluation results and updates the user's performance database. The output is the updated performance data.

[0503] Step 9:

[0504] Generating and displaying feedback

[0505] The server generates a feedback message and sends it to the terminal. The input is the evaluation result.

[0506] The device displays and provides a feedback message to the user. The output is the feedback message.

[0507] (Application Example 2)

[0508] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0509] Conventional customer service training systems only utilize dialogue based on speech recognition and natural language processing technologies, and are unable to recognize user emotions and provide appropriate responses and feedback based on them. Therefore, it was difficult to effectively train staff in the ability to understand user emotions and respond appropriately in real-world customer service situations. Furthermore, the inability to conduct real-time training using cutting-edge technologies such as smart glasses meant there was a lack of means to improve training efficiency.

[0510] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting the user's voice into text using a speech recognition engine, means for analyzing the text data using natural language processing technology to understand the user's intent, and means for providing emotional recognition engines that analyze the user's emotions and provide emotionally appropriate responses and feedback messages. This makes it possible to recognize the user's voice in real time, generate responses based on their intent and emotions, and perform training through smart glasses.

[0511] A "speech recognition engine" is a technology used to convert speech data into text data.

[0512] "Natural language processing technology" is an information processing technology that analyzes text data to understand the user's intent.

[0513] An "emotion recognition engine" is a technology that analyzes a user's emotions based on voice data and text data.

[0514] A "server device" is a computer system that performs various functions such as data processing and analysis, response generation, and feedback provision.

[0515] A "terminal device" is a user device that communicates with a server device and performs functions such as audio recording and playback, and interface display.

[0516] "Text data" refers to character information converted by a speech recognition engine.

[0517] A "response generation algorithm" is a program that creates an appropriate response based on the user's intentions and emotions.

[0518] "Smart glasses" are devices that, when worn by the user, provide visual information and interactive interfaces.

[0519] A "feedback message" is a message generated based on the user's training results and emotions, intended for evaluation and improvement.

[0520] A "persona" is a virtual character or role used to simulate a specific dialogue scenario or situation.

[0521] This invention is a system that combines voice dialogue AI technology and emotion recognition technology to provide real-time customer service training through smart glasses. The following describes embodiments for carrying out this invention.

[0522] System Configuration

[0523] 1. Server equipment

[0524] Speech recognition engine

[0525] Natural Language Processing Technology

[0526] Emotion recognition engine

[0527] Response generation algorithm

[0528] Evaluation and point calculation system

[0529] Database (user data, persona information, dialogue patterns, sentiment data)

[0530] 2. Terminal device

[0531] Smart Glasses

[0532] Microphone and speaker

[0533] Speech synthesis engine

[0534] Point and feedback display interface

[0535] Persona Selection Interface

[0536] Program details

[0537] The server device performs the following steps when a user uses a terminal device (smart glasses) to conduct customer service training.

[0538] 1. Voice recording and recognition

[0539] The server device receives the user's voice data collected by the terminal device and converts it into text data using a speech recognition engine.

[0540] 2. Natural Language Processing and Response Generation

[0541] The server device analyzes the converted text data using natural language processing technology to understand the user's intent. Next, it generates an appropriate response based on that intent using a response generation algorithm and sends it to the terminal device as text data.

[0542] 3. Emotion recognition and adjustment of responses

[0543] The server device passes voice and text data to the emotion recognition engine, which analyzes the user's emotions. Based on the analyzed emotion data, it adjusts its response and generates a more appropriate feedback message.

[0544] 4. Real-time training via smart glasses

[0545] The smart glasses, which are the terminal device, convert the received response into speech using a speech synthesis engine and play it back to the user. They also visually display the evaluation results to the user through a points and feedback display interface.

[0546] Devices and Software

[0547] The server device has a program installed to analyze the collected data and generate appropriate responses. The hardware and software used are as follows:

[0548] Speech recognition engine: Google Speech Recognition API

[0549] Natural Language Processing Techniques: Hugging Face's Transformers Library

[0550] Emotion Recognition Engine: Emotion Analysis Pipeline using the Transformers Library

[0551] Speech synthesis engine: Google Speech API

[0552] Smart glasses: Devices that provide visual and auditory information when worn by the user.

[0553] Specific example

[0554] When a store clerk speaks through smart glasses and says, "Could you tell me more about this product?", the system converts the speech into text and analyzes the clerk's emotions. If the clerk is excited, the system responds, "That's a great suggestion! Do you have any further questions?", and if the clerk is confused, it responds, "I'm sorry. I'll do my best to resolve the issue." The system always provides guidance that is appropriate to the clerk's emotions.

[0555] Example of a prompt:

[0556] User's utterance:

[0557] "Please tell me more about this product."

[0558] Example system response:

[0559] "That's a great suggestion! Do you have any further questions?"

[0560] User's emotional state: positive

[0561] Expected system response:

[0562] "That's a great suggestion! Do you have any further questions?"

[0563] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0564] Step 1:

[0565] The user puts on the smart glasses and starts training mode. The user selects the persona they want to train (e.g., "product description" or "customer complaint handling") through the persona selection interface. The device sends the selected persona information to the server.

[0566] Step 2:

[0567] The server retrieves appropriate dialogue patterns from the database based on the selected persona and prepares a training scenario. The server generates an initial question or situation description and sends the text data to the terminal.

[0568] Step 3:

[0569] The device converts the received text data into speech using a speech synthesis engine and plays it back to the user. The user then inputs the next action or response via voice.

[0570] Step 4:

[0571] The terminal records the user's voice input and sends it to the server as audio data. The server uses a speech recognition engine to convert the audio data into text data. The voice input is a specific question, such as "Please tell me more about this product."

[0572] Step 5:

[0573] The server analyzes the converted text data using natural language processing techniques to understand the user's intent, such as a specific request like "Please tell me the details of the product." Based on this analysis, the server generates the next appropriate response using a response generation algorithm.

[0574] Step 6:

[0575] The server analyzes the user's voice and text data using an emotion recognition engine to identify the user's emotional state. For example, it can determine whether the user is excited or confused.

[0576] Step 7:

[0577] The server adjusts its response based on the emotions identified by the emotion recognition engine, generating a more appropriate feedback message. For example, if the user is confused, it might respond with, "We're sorry. We'll do our best to resolve the issue." This response is then sent to the terminal as text data.

[0578] Step 8:

[0579] The device converts the received text data back into speech using a speech synthesis engine and plays it back to the user. This allows the user to receive feedback in real time.

[0580] Step 9:

[0581] The server passes the entire conversation record to the evaluation system, which scores the user's performance. Based on the evaluation results, points are calculated and the user's performance database is updated.

[0582] Step 10:

[0583] The device displays points and evaluation results to the user through a feedback display interface. For example, it provides evaluations using specific numerical values, such as "You earned 10 points in today's training."

[0584] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0585] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0586] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0587] [Second Embodiment]

[0588] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0589] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0590] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0591] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0592] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0593] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0594] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0595] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0596] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0597] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0598] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0599] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0600] This invention relates to a customer service training system that utilizes voice dialogue AI technology. This system aims to improve users' customer service skills through natural conversation using speech recognition and natural language processing (NLP). Specific embodiments of this invention are described below.

[0601] System Configuration

[0602] 1. Server equipment

[0603] Speech recognition engine

[0604] Natural Language Processing Engine

[0605] Response generation algorithm

[0606] Evaluation and point calculation system

[0607] Database (user data, persona information, dialogue patterns)

[0608] 2. Terminal device

[0609] Microphone and speaker

[0610] Speech synthesis engine

[0611] Point and feedback display interface

[0612] Persona Selection Interface

[0613] Specific steps for implementing the invention

[0614] Users of this system use a terminal device to begin customer service training. When a user starts a customer service scenario, the following processes occur.

[0615] 1. Voice recording and recognition

[0616] The (device) displays a recording start button, and the (user) presses the recording start button.

[0617] The (device) uses the microphone to record the (user's) voice.

[0618] The (device) sends the recorded audio data to the (server).

[0619] The server uses a speech recognition engine to convert the speech data into text data.

[0620] 2. Natural Language Processing and Response Generation

[0621] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[0622] The server generates an appropriate response based on the intent and sends that response as text data to the terminal.

[0623] The (device) converts text data into speech using a speech synthesis engine and plays the response back to the (user).

[0624] 3. Persona-based training

[0625] The user uses the terminal device's persona selection interface to select a specific customer type or persona (e.g., "Customer with a complaint").

[0626] The (device) sends the selected persona information to the (server).

[0627] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[0628] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[0629] The (device) converts the text data into speech and plays it back to the user.

[0630] As the training progresses, various persona-based dialogue scenarios unfold seamlessly.

[0631] 4. Evaluation and Feedback

[0632] The (terminal) records the user's speech in real time and sends it to the (server).

[0633] The server scores the user's utterances based on evaluation criteria.

[0634] The server calculates points based on the evaluation results and updates the user's performance database.

[0635] The server generates a feedback message and sends it to the terminal.

[0636] The (device) displays and provides a feedback message to the user.

[0637] Specific example

[0638] Example 1: Product Description Practice

[0639] The user selects a persona for the "product description" and starts the scenario.

[0640] The (device) started recording, and the AI ​​bot said, "Could you please explain this new product?"

[0641] The user says, "This product uses the latest technology..."

[0642] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[0643] The (device) plays the question aloud and continues the conversation.

[0644] Example 2: Practicing handling complaints

[0645] The user selects the "Complaint Handling" persona and starts the scenario.

[0646] The (device) started recording, and the AI ​​bot said, "This product arrived damaged, so could I request an exchange?"

[0647] The user says, "I am very sorry, I will take care of it immediately."

[0648] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[0649] The (device) plays the question aloud and continues the conversation.

[0650] Thus, this system is designed to allow users to efficiently acquire the skills necessary to handle a wide range of scenarios while actually engaging in dialogue.

[0651] The following describes the processing flow.

[0652] Utilization of voice-activated AI bots

[0653] Step 1:

[0654] The user presses the recording start button on the terminal device.

[0655] The (device) uses the microphone to record the user's voice.

[0656] Step 2:

[0657] The (device) saves the recorded audio data to a buffer.

[0658] The (terminal) sends the audio data in the buffer to the (server).

[0659] Step 3:

[0660] The server starts the speech recognition engine and converts the received speech data into text data.

[0661] The server generates the text data of the conversion result.

[0662] Step 4:

[0663] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[0664] The server generates a response based on the intended meaning.

[0665] Step 5:

[0666] The server generates a response as text data and sends it to the terminal.

[0667] The text data received by the terminal is converted into speech using a speech synthesis engine.

[0668] The (device) plays the converted audio through the speaker.

[0669] Persona-based training

[0670] Step 1:

[0671] The user uses the terminal device's interface to select a specific customer type or persona.

[0672] The (device) sends the selected persona information to the (server).

[0673] Step 2:

[0674] The server processes the persona information it receives.

[0675] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[0676] Step 3:

[0677] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[0678] The (device) receives text data, converts it into speech, and plays it back to the (user).

[0679] Role-playing progress evaluation and point system

[0680] Step 1:

[0681] The (device) records the user's speech in real time.

[0682] The (terminal) sends the recorded speech data to the (server).

[0683] Step 2:

[0684] The server analyzes the received speech data and evaluates its appropriateness and timing based on pre-set evaluation criteria.

[0685] The server calculates a score for each evaluation item.

[0686] Step 3:

[0687] The server calculates points based on the evaluation results.

[0688] The server adds points and updates the user's performance data.

[0689] Step 4:

[0690] The server sends the generated points and performance information to the terminal.

[0691] The terminal displays the points and performance information it has received and provides it to the user.

[0692] Feedback and improvement loop

[0693] Step 1:

[0694] The server re-evaluates the user's utterances and generates specific feedback.

[0695] The server sends the generated feedback message to the terminal as text data.

[0696] Step 2:

[0697] The (device) displays the feedback message it received on the screen.

[0698] The user reviews the feedback message and understands the areas for improvement.

[0699] (Example 1)

[0700] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0701] In customer service training, it is crucial for users to receive practical training based on diverse scenarios in order to effectively improve their conversational skills. However, conventional systems have not effectively performed real-time speech recognition or natural language processing, making it difficult to accurately understand and respond to user intentions. Furthermore, methods for evaluating user speech and providing feedback were insufficient, making it difficult for users to objectively evaluate their own skills and continuously improve them. Therefore, a new system is needed to effectively improve users' customer service skills.

[0702] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0703] In this invention, the server includes speech recognition means for converting user speech into text data, natural language processing means for analyzing the text data and understanding the user's intent, and response generation means for generating a response based on the user's intent and providing the generated response as text data. This enables users to engage in real-time dialogue based on diverse scenarios and effectively improve their customer service skills. Furthermore, by including means for providing an interface for selecting persona information, means for preparing dialogue patterns based on the selected persona, and means for starting a scenario based on the dialogue pattern and generating a voice response corresponding to the scenario, users can receive practical training according to diverse scenarios. In addition, by including means for recording and evaluating the user's utterances, means for calculating and updating points based on the evaluation, and means for displaying points and performance, users can objectively evaluate their own skills and continuously improve them.

[0704] "Voice recognition means" refers to a device or technology that records a user's voice and converts it into text data.

[0705] "Natural language processing" refers to technologies that analyze text data and understand the user's intent.

[0706] "Response generation means" refers to a device or technology that generates an appropriate response based on the user's intent and provides the generated response as text data.

[0707] "Speech synthesis means" refers to a device or technology that converts text data into speech and plays it back to the user.

[0708] "Persona information" refers to character information selected by specific customer types or users.

[0709] A "dialogue pattern" is a set of dialogue sequences and response patterns prepared based on a specific scenario.

[0710] A "scenario" is a sequence of dialogues based on a specific situation or case.

[0711] "Means for recording user speech content" refers to a device or technology that records the voice spoken by a user and stores it as data.

[0712] "Evaluation criteria" refer to the standards and rules used to evaluate the content of a user's utterances, and scoring is performed based on these criteria.

[0713] "Means for calculating points" refers to a device or technology that quantifies the user's utterances based on evaluation criteria and calculates them as points.

[0714] "Means for updating points" refers to a device or technology that reflects the calculated points in the user's performance database and updates it to the latest state.

[0715] An "interface" is a user interface or input device that allows a user to interact with a system.

[0716] A "feedback tool" is a device or technology that provides users with evaluation results, areas for improvement, and other information to help them improve their own skills.

[0717] This invention is a customer service training system that utilizes a speech recognition engine and natural language processing (NLP) technology. The system includes three main components: a server, a terminal, and a user.

[0718] First, the server includes a speech recognition engine, a natural language processing engine, a response generation algorithm, an evaluation and point calculation system, and a database that stores user data, persona information, and dialogue patterns. For the speech recognition engine, for example, the Google Cloud Speech-to-Text API can be used. For natural language processing, technologies such as spaCy or BERT are recommended. The response generation algorithm generates appropriate responses based on the user's intent and provides them as text data. The server also performs evaluations based on the user's utterances, calculates points, and generates feedback.

[0719] Next, the device includes a microphone and speaker, a speech synthesis engine, a point and feedback display interface, and a persona selection interface. For the speech synthesis engine, for example, Amazon Polly or Google Text-to-Speech can be used. The device records the user's voice and sends it to the server in real time. The text data received from the server is converted into speech and played back to the user. Furthermore, the persona selection interface allows the user to select specific customer types or personas.

[0720] The user receives customer service training using a device. For example, if the user selects the "product description" persona, the device starts recording and asks a voice question such as, "Could you please explain this new product?" When the user says, "This product uses the latest technology...", the device records the voice and sends it to the server. The server uses a speech recognition engine to convert the voice to text, analyzes it with a natural language processing engine, and then generates an appropriate response, "Could you tell me how to use it specifically?", and sends it to the device. The device converts that text data back into voice and plays it back to the user.

[0721] Furthermore, user performance is evaluated and calculated as points. The evaluation criteria are based on factors such as the content, timing, and politeness of the speech. Along with the evaluation results, points and feedback are provided to the user. For example, feedback such as "Your response this time was excellent" might be displayed.

[0722] Example of a prompt:

[0723] "Select a persona for your product description and begin explaining the new product."

[0724] "Let's select a persona for handling complaints and ask them to exchange the product."

[0725] In this way, users can efficiently improve their practical customer service skills through real-time interactions.

[0726] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0727] Step 1: Record and recognize audio

[0728] Input: User's speech

[0729] Operation: The device displays a recording start button, and recording begins when the user presses it. For example, the user might say, "Please explain the new product."

[0730] Data processing: The device's microphone records the user's voice and generates audio data.

[0731] Output: Recorded audio data

[0732] Operation: The device sends the recorded data to the server, and the server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio data into text data.

[0733] Step 2: Natural Language Processing and Response Generation

[0734] Input: Text data (converted from user's speech)

[0735] Operation: The server uses a natural language processing (NLP) engine (e.g., spaCy or BERT) to analyze text data and understand the user's intent. For example, it might analyze a request like "Please explain the new product."

[0736] Data processing: The NLP engine analyzes text data to identify the user's intent.

[0737] Output: User intent (e.g., the user is seeking a product description)

[0738] Operation: The server's response generation algorithm generates an appropriate response based on the user's intent and sends the generated response as text data to the terminal. For example, a response such as "What are its features?" might be generated.

[0739] Step 3: Speech synthesis and response playback

[0740] Input: Text data of the response

[0741] Operation: The device converts the received text data into speech using a text-to-speech engine (e.g., Amazon Polly or Google Text-to-Speech).

[0742] Data processing: The speech synthesis engine converts text data into speech data.

[0743] Output: Audio data

[0744] Operation: The device plays audio data and communicates a response to the user. For example, it might play the audio "What are its features?".

[0745] Step 4: Persona-based training

[0746] Input: Persona information selected by the user

[0747] Operation: The user selects a specific customer type or persona using the device's persona selection interface. For example, they might select "Customers with complaints."

[0748] Data processing: The device sends the selected persona information to the server, and the server prepares dialogue patterns based on the persona information.

[0749] Output: Dialogue patterns based on the selected persona

[0750] Operation: The server generates a scenario based on the dialogue pattern and sends the initial question and situation description as text data to the terminal. The terminal converts the text data into speech and plays the scenario. For example, it plays a scenario such as, "This product arrived damaged, could I request a replacement?"

[0751] Step 5: Record and evaluate the content of the speech.

[0752] Input: User's utterance

[0753] Operation: The device records the user's speech in real time and sends it to the server.

[0754] Data processing: The server scores the user's speech based on evaluation criteria. For example, it evaluates elements such as the content of the speech, timing, and politeness.

[0755] Output: Evaluation results and points

[0756] Operation: The server calculates points based on the evaluation results and updates the points.

[0757] Step 6: Generating and providing feedback

[0758] Input: Evaluation results and points

[0759] Operation: The server generates a feedback message and sends it to the terminal to inform the user.

[0760] Data processing: The feedback message is generated as text data.

[0761] Output: Feedback message

[0762] Action: The device displays and provides a feedback message to the user. For example, the feedback might say, "The service you provided was excellent."

[0763] In this way, the system effectively trains users' customer service skills through a series of steps, and provides evaluation and feedback.

[0764] (Application Example 1)

[0765] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0766] Conventional customer service training systems use speech recognition and natural language processing technologies to interact with users, but they have the drawback of not being able to evaluate actions and speech during training in real time. Furthermore, they lacked training that took user actions into account and detailed feedback on speech content. As a result, there was a problem in that improvements in customer service skills in actual stores were not sufficiently achieved.

[0767] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0768] In this invention, the server includes means for converting the user's speech into text using a speech recognition engine; means for analyzing the text data using natural language processing technology to understand the user's intent; means for generating a response based on the intent and providing the generated response as text data; means for capturing the user's actions with a camera and evaluating the synchronized speech content and actions; means for generating a response based on the user's speech content using a generative AI model; means for providing the generated response to the user as evaluation and feedback; and means for storing the speech content and evaluation results in a database. This makes it possible to grasp the user's actions and speech content in real time and provide evaluation and feedback on the spot.

[0769] A "speech recognition engine" is a device or software that captures a user's speech as a digital signal and converts that speech into text data.

[0770] "Natural language processing technology" is a technology that analyzes text data, understands the user's intentions and emotions, and enables natural dialogue between humans and computers.

[0771] "Means for generating responses" refer to algorithms or systems that construct appropriate responses based on the user's intent and provide them as text or audio data.

[0772] A "server device" is a central processing unit that receives and processes data from multiple users and provides necessary information and services.

[0773] A "terminal device" is a device that a user directly operates, equipped with a microphone, speaker, display, etc., and functions in conjunction with a server.

[0774] A "camera" is a device that captures a user's actions and gestures in real time and transmits that information to a server as image data.

[0775] A "generative AI model" is an artificial intelligence model that learns from a large dataset and generates natural-sounding responses based on the user's utterances.

[0776] A "prompt sentence" is an input sentence used by a generative AI model to generate a dialogue response, and it is an instruction sentence that appropriately reflects the user's intent.

[0777] A "database" is an information management system that stores and manages information such as spoken content, evaluation results, and dialogue history, and allows for quick reference as needed.

[0778] "Feedback" refers to information provided in real time regarding the evaluation and areas for improvement of the user's speech and actions during training.

[0779] This invention relates to a customer service training system using smart glasses, which provides real-time feedback utilizing speech recognition, natural language processing, and generative AI models. The system consists of a server device, terminal devices, and a network connecting them. Specific embodiments for carrying out the invention are described below.

[0780] System Configuration

[0781] The system consists of the following elements:

[0782] 1. Server equipment:

[0783] Speech recognition engine: Software that converts a user's speech into text data.

[0784] Natural Language Processing (NLP) technology: Software that analyzes text data to understand user intent.

[0785] Generative AI Model: This is an artificial intelligence model that generates responses based on the user's utterances. Specifically, it uses the Hugging Face transformers library and the GPT-3.5 model.

[0786] Database: A system that stores and manages spoken content, evaluation results, and dialogue history.

[0787] Response generation algorithm: An algorithm that generates an appropriate response based on the results of natural language processing.

[0788] Evaluation and Point Calculation System: A system that evaluates users in real time based on their speech and actions and calculates points.

[0789] 2. Terminal device (smart glasses):

[0790] Camera and microphone: Devices for capturing user actions and voice in real time.

[0791] Text-to-speech engine: Software that converts text data received from a server into speech and plays it back. Specifically, it uses the gTTS library.

[0792] Feedback display interface: An interface for displaying evaluation results and areas for improvement to the user.

[0793] Persona Selection Interface: An interface for users to select training scenarios.

[0794] Operation of the invention

[0795] 1. Customer service training begins:

[0796] The user wears smart glasses and selects a training scenario using a persona selection interface. This sends persona information to the server.

[0797] 2. Speech recognition and motion capture:

[0798] The smart glasses' microphone records the user's voice, and the camera captures their movements. The recorded audio data is sent to a server and converted into text data by a speech recognition engine.

[0799] 3. Natural Language Processing and Response Generation:

[0800] The server analyzes the text data and uses a generative AI model to generate an appropriate response. The generated response is then sent to the terminal device as text data.

[0801] 4. Speech synthesis and playback:

[0802] The terminal device's speech synthesis engine converts text data into speech and plays it back to the user.

[0803] 5. Evaluation and Feedback:

[0804] The server evaluates the user's speech and actions in real time and generates feedback. The evaluation results are stored in a database, and the feedback is provided to the user through a terminal device.

[0805] Specific example

[0806] As a concrete example, let's explain the introduction of a new product. When a user introduces a new product, the system operates in the following order:

[0807] 1. Scenario Selection:

[0808] The user selects "Introduce new products".

[0809] 2. Audio and motion capture:

[0810] The smart glasses capture the user's voice and movements, and the server analyzes them.

[0811] 3. Response generation:

[0812] User: "This product is a convenient product that uses the latest technology."

[0813] Server: The generation AI model generates a response based on the prompt text and responds with "What specific technologies are being used?".

[0814] 4. Audio Playback:

[0815] The smart glasses play the response aloud.

[0816] 5. Evaluation and Feedback:

[0817] The server evaluates the user's speech in real time and provides feedback such as, "It would be good if you could explain the product's features in more detail."

[0818] Example of a prompt

[0819] "The user is describing a new product. Please generate a natural-sounding continuation of the conversation. User: 'This product is a convenient product that uses the latest technology.' Please generate the AI's subsequent response."

[0820] This invention allows staff working in physical stores to efficiently improve their customer service skills while receiving real-time evaluation of user actions and speech. Furthermore, the use of a generative AI model enhances the naturalness and diversity of the dialogue.

[0821] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0822] Step 1:

[0823] The server receives information about the training scenario selected by the user (e.g., "Introduction to a New Product") from the persona selection interface of the terminal device (smart glasses). The input data is the persona information selected by the user, and the server prepares dialogue patterns corresponding to that persona as output.

[0824] Step 2:

[0825] The device uses a microphone built into the smart glasses to record the user's speech in real time. The input data is the voice spoken by the user, and the output data is the digital signal of that voice. This digital signal is sent to a server.

[0826] Step 3:

[0827] The server analyzes the received audio data using a speech recognition engine and converts it into text data. The input data is a digital audio signal, and the output data is the text data obtained by analyzing that audio signal.

[0828] Step 4:

[0829] The server analyzes text data using natural language processing techniques to understand the user's intent. The input data is text data obtained by a speech recognition engine, and the output data is metadata containing the user's intent as a result of the analysis.

[0830] Step 5:

[0831] The server uses a generative AI model based on the user's intent to generate an appropriate response. The input data consists of metadata and prompts for the generative AI model, parsed using natural language processing techniques, while the output data is the text data of the generated response.

[0832] Step 6:

[0833] The server sends the generated response text data to the terminal device. The input data is the response text data, and the output is the transmission of the text data to the terminal device.

[0834] Step 7:

[0835] The terminal converts the received response text data into speech using a speech synthesis engine and plays it back to the user. The input data is text data received from the server, and the output data is speech generated by speech synthesis.

[0836] Step 8:

[0837] The device uses the smart glasses' camera to capture the user's movements and sends the video data to the server. The input data is the video captured by the camera, and the output data is the digital signal of that video.

[0838] Step 9:

[0839] The server evaluates the user's speech and motion data and generates feedback. The input data consists of audio and video data transmitted from the smart glasses, and the output data is a feedback message representing the evaluation result.

[0840] Step 10:

[0841] The server sends the generated feedback message to the terminal device and plays it back to the user either visually or audibly. The input data is the feedback message, and the output data is the transmission of the message to the terminal device and its display or audibility.

[0842] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0843] This invention relates to a customer service training system that combines voice dialogue AI technology and emotion recognition. This system recognizes the user's voice and understands the user's intentions based on natural language processing technology, as well as recognizing the user's emotions using an emotion engine. Based on this information, it generates appropriate responses to improve skills. Specific embodiments of this invention are described below.

[0844] System Configuration

[0845] 1. Server equipment

[0846] Speech recognition engine

[0847] Natural Language Processing Engine

[0848] Emotional Engine

[0849] Response generation algorithm

[0850] Evaluation and point calculation system

[0851] Database (user data, persona information, dialogue patterns, sentiment data)

[0852] 2. Terminal device

[0853] Microphone and speaker

[0854] Speech synthesis engine

[0855] Point and feedback display interface

[0856] Persona Selection Interface

[0857] Specific steps for implementing the invention

[0858] Users of this system use a terminal device to begin customer service training, including emotion recognition. When a user starts a scenario, the following processes occur:

[0859] 1. Voice recording and recognition

[0860] The user presses the recording start button on the terminal device.

[0861] The (device) uses the microphone to record the (user's) voice.

[0862] The (device) sends the recorded audio data to the (server).

[0863] The server uses a speech recognition engine to convert the speech data into text data.

[0864] 2. Natural Language Processing and Response Generation

[0865] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent.

[0866] The server generates an appropriate response based on the intent and sends that response as text data to the terminal.

[0867] The (device) converts the text data into speech using a speech synthesis engine and plays the response back to the (user).

[0868] 3. Emotion recognition

[0869] The server passes voice and text data to the emotion engine, which analyzes the user's emotions.

[0870] The server adjusts its response based on emotional data to generate more appropriate dialogue.

[0871] 4. Persona-based training

[0872] The user selects a persona on the terminal device (e.g., "Customer with a complaint").

[0873] The (device) sends the selected persona information to the (server).

[0874] The server references persona-based dialogue patterns and prepares the scenario.

[0875] The server generates the first question according to the scenario and sends it to the terminal as text data.

[0876] The (device) converts the text data into speech and plays it back to the user.

[0877] 5. Evaluation and Point System

[0878] The (terminal) records the user's speech in real time and sends it to the (server).

[0879] The server scores the speech content and emotional data based on evaluation criteria.

[0880] The server calculates points based on the evaluation results and updates the user's performance database.

[0881] The server generates a feedback message and sends it to the terminal.

[0882] The (device) displays and provides a feedback message to the user.

[0883] Specific example

[0884] Example 1: Product Description Practice

[0885] The user selects a persona for the "product description" and starts the scenario.

[0886] The (device) started recording, and the AI ​​bot said, "Could you please explain this new product?"

[0887] The user says, "This product uses the latest technology..."

[0888] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[0889] The server recognizes the user's emotions from their voice and generates feedback that corresponds to those emotions.

[0890] The (device) plays back questions and feedback via audio, continuing the conversation.

[0891] Example 2: Practicing handling complaints

[0892] The user selects the "Complaint Handling" persona and starts the scenario.

[0893] The (device) started recording, and the AI ​​bot said, "This product arrived damaged, so could I request an exchange?"

[0894] The user says, "I am very sorry, I will take care of it immediately."

[0895] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[0896] The server uses an emotion engine to recognize the user's anger state and generates an additional response to calm them down.

[0897] The (device) plays the question and additional responses aloud, continuing the conversation.

[0898] This system is designed to allow users to efficiently acquire the skills necessary to handle a wide range of scenarios while engaging in actual conversations. Furthermore, the emotion recognition function enables the creation of even more realistic conversation scenarios.

[0899] The following describes the processing flow.

[0900] Utilization of voice-activated AI bots and emotion recognition

[0901] Step 1:

[0902] The user presses the recording start button on the terminal device.

[0903] The (device) uses the microphone to record the user's voice.

[0904] Step 2:

[0905] The (device) saves the recorded audio data to a buffer.

[0906] The (terminal) sends the audio data in the buffer to the (server).

[0907] Step 3:

[0908] The server starts the speech recognition engine and converts the received speech data into text data.

[0909] The server generates the text data of the conversion result.

[0910] Step 4:

[0911] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[0912] The server generates a response based on the intended meaning.

[0913] Step 5:

[0914] The server passes voice and text data to the emotion engine, which analyzes the user's emotions.

[0915] The server adjusts its response based on the emotion recognition results.

[0916] Step 6:

[0917] The server generates a pre-arranged response as text data and sends it to the terminal.

[0918] The text data received by the terminal is converted into speech using a speech synthesis engine.

[0919] The (device) plays the converted audio through the speaker.

[0920] Persona-based training

[0921] Step 1:

[0922] The user uses the terminal device's interface to select a specific customer type or persona.

[0923] The (device) sends the selected persona information to the (server).

[0924] Step 2:

[0925] The server processes the persona information it receives.

[0926] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[0927] Step 3:

[0928] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[0929] The (device) receives text data, converts it into speech, and plays it back to the (user).

[0930] Role-playing progress evaluation and point system

[0931] Step 1:

[0932] The (device) records the user's speech in real time.

[0933] The (terminal) sends the recorded speech data to the (server).

[0934] Step 2:

[0935] The server analyzes the received speech data and evaluates its appropriateness and timing based on pre-set evaluation criteria.

[0936] The server calculates a score for each evaluation item based on speech data and emotion recognition data.

[0937] Step 3:

[0938] The server calculates points based on the evaluation results.

[0939] The server adds points and updates the user's performance data.

[0940] Step 4:

[0941] The server sends the generated points and performance information to the terminal.

[0942] The terminal displays the points and performance information it has received and provides it to the user.

[0943] Feedback and improvement loop

[0944] Step 1:

[0945] The server re-evaluates the user's utterance and generates specific feedback.

[0946] The server sends the generated feedback message to the terminal as text data.

[0947] Step 2:

[0948] The (device) displays the feedback message it received on the screen.

[0949] The user reviews the feedback message and understands the areas for improvement.

[0950] (Example 2)

[0951] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0952] Conventional customer service training systems could understand user intent and generate responses using speech recognition and natural language processing technologies, but they struggled to respond while considering user emotions. Furthermore, there were few means to evaluate the effectiveness of the training, making it difficult to efficiently improve user performance and skills. Additionally, training based on specific scenarios was limited, making it difficult to simulate real-world customer service situations.

[0953] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0954] In this invention, the server includes means for converting user speech to text using a speech recognition engine, means for analyzing text data using natural language processing technology to understand user intent, means for generating a response based on intent and providing the generated response as text data, means for converting text data to speech and playing it back to the user, means for analyzing speech and text data to recognize user emotions, means for adjusting responses based on emotions to generate more appropriate dialogue, means for providing an interface for selecting persona information, means for preparing dialogue patterns based on the selected persona, means for starting a scenario based on the dialogue pattern and generating a speech response corresponding to the scenario, means for recording and evaluating user utterances, means for calculating points based on evaluation and updating points, and means for displaying points and performance. This enables the provision of appropriate responses that take emotions into account, improves the quality of dialogue, and allows for evaluation of user skills and confirmation of performance. Furthermore, by realizing training based on specific scenarios, it becomes possible to simulate situations that closely resemble actual customer service scenarios.

[0955] A "speech recognition engine" is a technology that takes speech data as input and converts it into text data.

[0956] "Natural language processing" is a technology that analyzes text data, understands human language, and generates appropriate responses.

[0957] A "server device" is a computer system that provides data processing and storage functions and operates in conjunction with client devices.

[0958] A "terminal device" is a device that a user directly operates and that provides interface functions while communicating with a server.

[0959] An "emotion engine" is a technology that analyzes voice and text data to estimate a user's emotional state.

[0960] A "persona" is a fictional character based on a specific scenario or user group, and is the subject of simulation in a training system.

[0961] A "dialogue pattern" is a definition of the expected sequence of questions and answers in a particular scenario.

[0962] "Evaluation criteria" are standards or standards used to evaluate user performance based on the content of user utterances and emotional state.

[0963] "Feedback" refers to evaluation comments and advice provided regarding a user's performance.

[0964] System Overview

[0965] This invention relates to a customer service training system that combines voice dialogue AI technology and emotion recognition. This system recognizes the user's voice and understands their intentions based on natural language processing technology, as well as recognizing the user's emotions using an emotion engine. Based on this information, it generates appropriate responses to improve skills.

[0966] Hardware and software configuration

[0967] This system consists of the following components:

[0968] Server equipment

[0969] Speech recognition engine: Converts speech data into text (e.g., Google Cloud Speech-to-Text)

[0970] Natural language processing engine: Analyzes text data and understands user intent (e.g., GPT-3).

[0971] Emotion engine: Analyzes user emotions based on voice and text data (e.g., Microsoft Azure Emotion API)

[0972] Response generation algorithm: Generates appropriate responses based on the user's intent and emotions.

[0973] Evaluation and point calculation system: Evaluates the user's utterances and calculates points.

[0974] Database: Stores user data, persona information, dialogue patterns, and sentiment data.

[0975] Terminal device

[0976] Microphone and speaker: Used for audio input and output.

[0977] Speech synthesis engine: Converts text data into speech (e.g., Amazon Polly)

[0978] Point and feedback display interface: Displays points and feedback to the user.

[0979] Persona Selection Interface: An interface for users to select a persona.

[0980] Specific implementations of the system

[0981] Users of this system use a terminal device to begin customer service training, including emotion recognition. Specific embodiments of the present invention are shown below.

[0982] Voice recording and recognition

[0983] The user presses the recording start button on the terminal device.

[0984] The device uses the microphone to record the user's voice.

[0985] The device sends the recorded audio data to the server.

[0986] The server uses a speech recognition engine to convert the audio data into text data.

[0987] Natural language processing and response generation

[0988] The server analyzes the text data using a natural language processing engine to understand the user's intent.

[0989] The server generates an appropriate response based on the user's intent and sends that response as text data to the terminal.

[0990] The device converts text data into speech using a speech synthesis engine and plays the response back to the user.

[0991] emotion recognition

[0992] The server passes voice and text data to the emotion engine, which then analyzes the user's emotions.

[0993] The server adjusts its response based on emotional data to generate more appropriate dialogue.

[0994] Persona-based training

[0995] The user selects a persona (e.g., "a customer with a complaint") on the terminal device.

[0996] The device sends the selected persona information to the server.

[0997] The server references persona-based dialogue patterns and prepares the scenario.

[0998] The server generates an initial question based on the scenario and sends it to the terminal as text data.

[0999] The device converts text data into speech and plays it back to the user.

[1000] Rating and point system

[1001] The device records the user's speech in real time and sends it to the server.

[1002] The server scores the spoken content and emotional data based on evaluation criteria.

[1003] The server calculates points based on the evaluation results and updates the user's performance database.

[1004] The server generates a feedback message and sends it to the terminal.

[1005] The device displays and provides a feedback message to the user.

[1006] Specific usage examples

[1007] Product description practice

[1008] The user selects a persona for the "product description" and starts the scenario.

[1009] The device starts recording and says, "Could you please explain this new product?"

[1010] The user says, "This product uses the latest technology..."

[1011] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[1012] The server recognizes emotions from the user's voice and generates emotionally appropriate feedback.

[1013] The device plays back questions and feedback via voice, continuing the conversation.

[1014] Practice in handling complaints

[1015] The user selects the "complaint handling" persona and starts the scenario.

[1016] The device starts recording and says, "This product arrived damaged, so could I request a replacement?"

[1017] The user says, "I am very sorry, I will take care of it immediately."

[1018] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[1019] The server uses an emotion engine to recognize the user's anger level and generates additional responses to calm them down.

[1020] The device plays back the question and any additional responses via audio, continuing the conversation.

[1021] Examples of prompts for generative AI models

[1022] "Please provide a detailed explanation of the new product."

[1023] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1024] Step 1:

[1025] Audio recording and transfer

[1026] The user presses the recording start button on the terminal device. The input is the user's voice.

[1027] The device uses the microphone to record the user's voice. The output is the recorded audio data.

[1028] The device sends the recorded audio data to the server. The audio data is transferred from the device to the server.

[1029] Step 2:

[1030] Text conversion of audio data

[1031] The server uses a speech recognition engine to convert audio data into text data. The input is recorded audio data.

[1032] A speech recognition engine (such as Google Cloud Speech-to-Text) analyzes the audio data and converts it into text data. The output is text data.

[1033] Step 3:

[1034] Natural language processing and intent analysis

[1035] The server analyzes text data using a natural language processing engine. The input is text data.

[1036] A natural language processing engine (such as GPT-3) analyzes text data to understand the user's intent. The output is data that reflects the user's intent.

[1037] Step 4:

[1038] Response generation

[1039] The server generates an appropriate response based on the user's intent. The input is parsed intent data.

[1040] The response generation algorithm creates a text response tailored to the user's intent. The output is the text data of the response.

[1041] Step 5:

[1042] Speech and playback of responses

[1043] The server sends the generated text data to the terminal. The input is the response text data.

[1044] The device converts text data into speech using a speech synthesis engine (such as Amazon Polly). The output is the generated speech data.

[1045] The terminal plays the generated audio to the user. The audio data is provided to the user through the audio output device.

[1046] Step 6:

[1047] emotion recognition

[1048] The server passes audio and text data to the emotion engine. The input consists of audio and text data.

[1049] An emotion engine (such as the Microsoft Azure Emotion API) analyzes the data and recognizes the user's emotions. The output is emotion data.

[1050] The server adjusts its response based on sentiment data and generates a more appropriate dialogue. Adjusted response data is then generated.

[1051] Step 7:

[1052] Persona-based training begins.

[1053] The user selects a persona (e.g., "Customer with a complaint") on the terminal device. Persona information is then entered.

[1054] The device sends the selected persona information to the server. The input is the selected persona data.

[1055] The server references persona-based dialogue patterns and prepares a scenario. The output is scenario data.

[1056] The server generates the first question based on the scenario and sends it to the terminal as text data. The input is the scenario data.

[1057] The device converts text data into speech and plays it back to the user. The output is audio data.

[1058] Step 8:

[1059] Recording and evaluation of spoken content

[1060] The terminal records the user's speech in real time and sends it to the server. The input is the user's speech.

[1061] The server scores the utterance content and emotional data based on evaluation criteria. The input consists of the utterance content and emotional data.

[1062] The server calculates points based on the evaluation results and updates the user's performance database. The output is the updated performance data.

[1063] Step 9:

[1064] Generating and displaying feedback

[1065] The server generates a feedback message and sends it to the terminal. The input is the evaluation result.

[1066] The device displays and provides a feedback message to the user. The output is the feedback message.

[1067] (Application Example 2)

[1068] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1069] Conventional customer service training systems only utilize dialogue based on speech recognition and natural language processing technologies, and are unable to recognize user emotions and provide appropriate responses and feedback based on them. Therefore, it was difficult to effectively train staff in the ability to understand user emotions and respond appropriately in real-world customer service situations. Furthermore, the inability to conduct real-time training using cutting-edge technologies such as smart glasses meant there was a lack of means to improve training efficiency.

[1070] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting the user's voice into text using a speech recognition engine, means for analyzing the text data using natural language processing technology to understand the user's intent, and means for providing emotional recognition engines that analyze the user's emotions and provide emotionally appropriate responses and feedback messages. This makes it possible to recognize the user's voice in real time, generate responses based on their intent and emotions, and perform training through smart glasses.

[1071] A "speech recognition engine" is a technology used to convert speech data into text data.

[1072] "Natural language processing technology" is an information processing technology that analyzes text data to understand the user's intent.

[1073] An "emotion recognition engine" is a technology that analyzes a user's emotions based on voice data and text data.

[1074] A "server device" is a computer system that performs various functions such as data processing and analysis, response generation, and feedback provision.

[1075] A "terminal device" is a user device that communicates with a server device and performs functions such as audio recording and playback, and interface display.

[1076] "Text data" refers to character information converted by a speech recognition engine.

[1077] A "response generation algorithm" is a program that creates an appropriate response based on the user's intentions and emotions.

[1078] "Smart glasses" are devices that, when worn by the user, provide visual information and interactive interfaces.

[1079] A "feedback message" is a message generated based on the user's training results and emotions, intended for evaluation and improvement.

[1080] A "persona" is a virtual character or role used to simulate a specific dialogue scenario or situation.

[1081] This invention is a system that combines voice dialogue AI technology and emotion recognition technology to provide real-time customer service training through smart glasses. The following describes embodiments for carrying out this invention.

[1082] System Configuration

[1083] 1. Server equipment

[1084] Speech recognition engine

[1085] Natural Language Processing Technology

[1086] Emotion recognition engine

[1087] Response generation algorithm

[1088] Evaluation and point calculation system

[1089] Database (user data, persona information, dialogue patterns, sentiment data)

[1090] 2. Terminal device

[1091] Smart Glasses

[1092] Microphone and speaker

[1093] Speech synthesis engine

[1094] Point and feedback display interface

[1095] Persona Selection Interface

[1096] Program details

[1097] The server device performs the following steps when a user uses a terminal device (smart glasses) to conduct customer service training.

[1098] 1. Voice recording and recognition

[1099] The server device receives the user's voice data collected by the terminal device and converts it into text data using a speech recognition engine.

[1100] 2. Natural Language Processing and Response Generation

[1101] The server device analyzes the converted text data using natural language processing technology to understand the user's intent. Next, it generates an appropriate response based on that intent using a response generation algorithm and sends it to the terminal device as text data.

[1102] 3. Emotion recognition and adjustment of responses

[1103] The server device passes voice and text data to the emotion recognition engine, which analyzes the user's emotions. Based on the analyzed emotion data, it adjusts its response and generates a more appropriate feedback message.

[1104] 4. Real-time training via smart glasses

[1105] The smart glasses, which are the terminal device, convert the received response into speech using a speech synthesis engine and play it back to the user. They also visually display the evaluation results to the user through a points and feedback display interface.

[1106] Devices and Software

[1107] The server device has a program installed to analyze the collected data and generate appropriate responses. The hardware and software used are as follows:

[1108] Speech recognition engine: Google Speech Recognition API

[1109] Natural Language Processing Techniques: Hugging Face's Transformers Library

[1110] Emotion Recognition Engine: Emotion Analysis Pipeline using the Transformers Library

[1111] Speech synthesis engine: Google Speech API

[1112] Smart glasses: Devices that provide visual and auditory information when worn by the user.

[1113] Specific example

[1114] When a store clerk speaks through smart glasses and says, "Could you tell me more about this product?", the system converts the speech into text and analyzes the clerk's emotions. If the clerk is excited, the system responds, "That's a great suggestion! Do you have any further questions?", and if the clerk is confused, it responds, "I'm sorry. I'll do my best to resolve the issue." The system always provides guidance that is appropriate to the clerk's emotions.

[1115] Example of a prompt:

[1116] User's utterance:

[1117] "Please tell me more about this product."

[1118] Example system response:

[1119] "That's a great suggestion! Do you have any further questions?"

[1120] User's emotional state: positive

[1121] Expected system response:

[1122] "That's a great suggestion! Do you have any further questions?"

[1123] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1124] Step 1:

[1125] The user puts on the smart glasses and starts training mode. The user selects the persona they want to train (e.g., "product description" or "customer complaint handling") through the persona selection interface. The device sends the selected persona information to the server.

[1126] Step 2:

[1127] The server retrieves appropriate dialogue patterns from the database based on the selected persona and prepares a training scenario. The server generates an initial question or situation description and sends the text data to the terminal.

[1128] Step 3:

[1129] The device converts the received text data into speech using a speech synthesis engine and plays it back to the user. The user then inputs the next action or response via voice.

[1130] Step 4:

[1131] The terminal records the user's voice input and sends it to the server as audio data. The server uses a speech recognition engine to convert the audio data into text data. The voice input is a specific question, such as "Please tell me more about this product."

[1132] Step 5:

[1133] The server analyzes the converted text data using natural language processing techniques to understand the user's intent, such as a specific request like "Please tell me the details of the product." Based on this analysis, the server generates the next appropriate response using a response generation algorithm.

[1134] Step 6:

[1135] The server analyzes the user's voice and text data using an emotion recognition engine to identify the user's emotional state. For example, it can determine whether the user is excited or confused.

[1136] Step 7:

[1137] The server adjusts its response based on the emotions identified by the emotion recognition engine, generating a more appropriate feedback message. For example, if the user is confused, it might respond with, "We're sorry. We'll do our best to resolve the issue." This response is then sent to the terminal as text data.

[1138] Step 8:

[1139] The device converts the received text data back into speech using a speech synthesis engine and plays it back to the user. This allows the user to receive feedback in real time.

[1140] Step 9:

[1141] The server passes the entire conversation record to the evaluation system, which scores the user's performance. Based on the evaluation results, points are calculated and the user's performance database is updated.

[1142] Step 10:

[1143] The device displays points and evaluation results to the user through a feedback display interface. For example, it provides evaluations using specific numerical values, such as "You earned 10 points in today's training."

[1144] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1145] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1146] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1147] [Third Embodiment]

[1148] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1149] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1150] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1151] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1152] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1153] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1154] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1155] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1156] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1157] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1158] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1159] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1160] This invention relates to a customer service training system that utilizes voice dialogue AI technology. This system aims to improve users' customer service skills through natural conversation using speech recognition and natural language processing (NLP). Specific embodiments of this invention are described below.

[1161] System Configuration

[1162] 1. Server equipment

[1163] Speech recognition engine

[1164] Natural Language Processing Engine

[1165] Response generation algorithm

[1166] Evaluation and point calculation system

[1167] Database (user data, persona information, dialogue patterns)

[1168] 2. Terminal device

[1169] Microphone and speaker

[1170] Speech synthesis engine

[1171] Point and feedback display interface

[1172] Persona Selection Interface

[1173] Specific steps for implementing the invention

[1174] Users of this system use a terminal device to begin customer service training. When a user starts a customer service scenario, the following processes occur.

[1175] 1. Voice recording and recognition

[1176] The (device) displays a recording start button, and the (user) presses the recording start button.

[1177] The (device) uses the microphone to record the (user's) voice.

[1178] The (device) sends the recorded audio data to the (server).

[1179] The server uses a speech recognition engine to convert the speech data into text data.

[1180] 2. Natural Language Processing and Response Generation

[1181] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[1182] The server generates an appropriate response based on the intent and sends that response as text data to the terminal.

[1183] The (device) converts text data into speech using a speech synthesis engine and plays the response back to the (user).

[1184] 3. Persona-based training

[1185] The user uses the terminal device's persona selection interface to select a specific customer type or persona (e.g., "Customer with a complaint").

[1186] The (device) sends the selected persona information to the (server).

[1187] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[1188] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[1189] The (device) converts the text data into speech and plays it back to the user.

[1190] As the training progresses, various persona-based dialogue scenarios unfold seamlessly.

[1191] 4. Evaluation and Feedback

[1192] The (terminal) records the user's speech in real time and sends it to the (server).

[1193] The server scores the user's utterances based on evaluation criteria.

[1194] The server calculates points based on the evaluation results and updates the user's performance database.

[1195] The server generates a feedback message and sends it to the terminal.

[1196] The (device) displays and provides a feedback message to the user.

[1197] Specific example

[1198] Example 1: Product Description Practice

[1199] The user selects a persona for the "product description" and starts the scenario.

[1200] The (device) started recording, and the AI ​​bot said, "Could you please explain this new product?"

[1201] The user says, "This product uses the latest technology..."

[1202] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[1203] The (device) plays the question aloud and continues the conversation.

[1204] Example 2: Practicing handling complaints

[1205] The user selects the "Complaint Handling" persona and starts the scenario.

[1206] The (device) started recording, and the AI ​​bot said, "This product arrived damaged, so could I request an exchange?"

[1207] The user says, "I am very sorry, I will take care of it immediately."

[1208] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[1209] The (device) plays the question aloud and continues the conversation.

[1210] Thus, this system is designed to allow users to efficiently acquire the skills necessary to handle a wide range of scenarios while actually engaging in dialogue.

[1211] The following describes the processing flow.

[1212] Utilization of voice-activated AI bots

[1213] Step 1:

[1214] The user presses the recording start button on the terminal device.

[1215] The (device) uses the microphone to record the user's voice.

[1216] Step 2:

[1217] The (device) saves the recorded audio data to a buffer.

[1218] The (terminal) sends the audio data in the buffer to the (server).

[1219] Step 3:

[1220] The server starts the speech recognition engine and converts the received speech data into text data.

[1221] The server generates the text data of the conversion result.

[1222] Step 4:

[1223] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[1224] The server generates a response based on the intended meaning.

[1225] Step 5:

[1226] The server generates a response as text data and sends it to the terminal.

[1227] The text data received by the terminal is converted into speech using a speech synthesis engine.

[1228] The (device) plays the converted audio through the speaker.

[1229] Persona-based training

[1230] Step 1:

[1231] The user uses the terminal device's interface to select a specific customer type or persona.

[1232] The (device) sends the selected persona information to the (server).

[1233] Step 2:

[1234] The server processes the persona information it receives.

[1235] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[1236] Step 3:

[1237] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[1238] The (device) receives text data, converts it into speech, and plays it back to the (user).

[1239] Role-playing progress evaluation and point system

[1240] Step 1:

[1241] The (device) records the user's speech in real time.

[1242] The (terminal) sends the recorded speech data to the (server).

[1243] Step 2:

[1244] The server analyzes the received speech data and evaluates its appropriateness and timing based on pre-set evaluation criteria.

[1245] The server calculates a score for each evaluation item.

[1246] Step 3:

[1247] The server calculates points based on the evaluation results.

[1248] The server adds points and updates the user's performance data.

[1249] Step 4:

[1250] The server sends the generated points and performance information to the terminal.

[1251] The terminal displays the points and performance information it has received and provides it to the user.

[1252] Feedback and improvement loop

[1253] Step 1:

[1254] The server re-evaluates the user's utterances and generates specific feedback.

[1255] The server sends the generated feedback message to the terminal as text data.

[1256] Step 2:

[1257] The (device) displays the feedback message it received on the screen.

[1258] The user reviews the feedback message and understands the areas for improvement.

[1259] (Example 1)

[1260] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1261] In customer service training, it is crucial for users to receive practical training based on diverse scenarios in order to effectively improve their conversational skills. However, conventional systems have not effectively performed real-time speech recognition or natural language processing, making it difficult to accurately understand and respond to user intentions. Furthermore, methods for evaluating user speech and providing feedback were insufficient, making it difficult for users to objectively evaluate their own skills and continuously improve them. Therefore, a new system is needed to effectively improve users' customer service skills.

[1262] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1263] In this invention, the server includes speech recognition means for converting user speech into text data, natural language processing means for analyzing the text data and understanding the user's intent, and response generation means for generating a response based on the user's intent and providing the generated response as text data. This enables users to engage in real-time dialogue based on diverse scenarios and effectively improve their customer service skills. Furthermore, by including means for providing an interface for selecting persona information, means for preparing dialogue patterns based on the selected persona, and means for starting a scenario based on the dialogue pattern and generating a voice response corresponding to the scenario, users can receive practical training according to diverse scenarios. In addition, by including means for recording and evaluating the user's utterances, means for calculating and updating points based on the evaluation, and means for displaying points and performance, users can objectively evaluate their own skills and continuously improve them.

[1264] "Voice recognition means" refers to a device or technology that records a user's voice and converts it into text data.

[1265] "Natural language processing" refers to technologies that analyze text data and understand the user's intent.

[1266] "Response generation means" refers to a device or technology that generates an appropriate response based on the user's intent and provides the generated response as text data.

[1267] "Speech synthesis means" refers to a device or technology that converts text data into speech and plays it back to the user.

[1268] "Persona information" refers to character information selected by specific customer types or users.

[1269] A "dialogue pattern" is a set of dialogue sequences and response patterns prepared based on a specific scenario.

[1270] A "scenario" is a sequence of dialogues based on a specific situation or case.

[1271] "Means for recording user speech content" refers to a device or technology that records the voice spoken by a user and stores it as data.

[1272] "Evaluation criteria" refer to the standards and rules used to evaluate the content of a user's utterances, and scoring is performed based on these criteria.

[1273] "Means for calculating points" refers to a device or technology that quantifies the user's utterances based on evaluation criteria and calculates them as points.

[1274] "Means for updating points" refers to a device or technology that reflects the calculated points in the user's performance database and updates it to the latest state.

[1275] An "interface" is a user interface or input device that allows a user to interact with a system.

[1276] A "feedback tool" is a device or technology that provides users with evaluation results, areas for improvement, and other information to help them improve their own skills.

[1277] This invention is a customer service training system that utilizes a speech recognition engine and natural language processing (NLP) technology. The system includes three main components: a server, a terminal, and a user.

[1278] First, the server includes a speech recognition engine, a natural language processing engine, a response generation algorithm, an evaluation and point calculation system, and a database that stores user data, persona information, and dialogue patterns. For the speech recognition engine, for example, the Google Cloud Speech-to-Text API can be used. For natural language processing, technologies such as spaCy or BERT are recommended. The response generation algorithm generates appropriate responses based on the user's intent and provides them as text data. The server also performs evaluations based on the user's utterances, calculates points, and generates feedback.

[1279] Next, the device includes a microphone and speaker, a speech synthesis engine, a point and feedback display interface, and a persona selection interface. For the speech synthesis engine, for example, Amazon Polly or Google Text-to-Speech can be used. The device records the user's voice and sends it to the server in real time. The text data received from the server is converted into speech and played back to the user. Furthermore, the persona selection interface allows the user to select specific customer types or personas.

[1280] The user receives customer service training using a device. For example, if the user selects the "product description" persona, the device starts recording and asks a voice question such as, "Could you please explain this new product?" When the user says, "This product uses the latest technology...", the device records the voice and sends it to the server. The server uses a speech recognition engine to convert the voice to text, analyzes it with a natural language processing engine, and then generates an appropriate response, "Could you tell me how to use it specifically?", and sends it to the device. The device converts that text data back into voice and plays it back to the user.

[1281] Furthermore, user performance is evaluated and calculated as points. The evaluation criteria are based on factors such as the content, timing, and politeness of the speech. Along with the evaluation results, points and feedback are provided to the user. For example, feedback such as "Your response this time was excellent" might be displayed.

[1282] Example of a prompt:

[1283] "Select a persona for your product description and begin explaining the new product."

[1284] "Let's select a persona for handling complaints and ask them to exchange the product."

[1285] In this way, users can efficiently improve their practical customer service skills through real-time interactions.

[1286] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1287] Step 1: Record and recognize audio

[1288] Input: User's speech

[1289] Operation: The device displays a recording start button, and recording begins when the user presses it. For example, the user might say, "Please explain the new product."

[1290] Data processing: The device's microphone records the user's voice and generates audio data.

[1291] Output: Recorded audio data

[1292] Operation: The device sends the recorded data to the server, and the server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio data into text data.

[1293] Step 2: Natural Language Processing and Response Generation

[1294] Input: Text data (converted from user's speech)

[1295] Operation: The server uses a natural language processing (NLP) engine (e.g., spaCy or BERT) to analyze text data and understand the user's intent. For example, it might analyze a request like "Please explain the new product."

[1296] Data processing: The NLP engine analyzes text data to identify the user's intent.

[1297] Output: User intent (e.g., the user is seeking a product description)

[1298] Operation: The server's response generation algorithm generates an appropriate response based on the user's intent and sends the generated response as text data to the terminal. For example, a response such as "What are its features?" might be generated.

[1299] Step 3: Speech synthesis and response playback

[1300] Input: Text data of the response

[1301] Operation: The device converts the received text data into speech using a text-to-speech engine (e.g., Amazon Polly or Google Text-to-Speech).

[1302] Data processing: The speech synthesis engine converts text data into speech data.

[1303] Output: Audio data

[1304] Operation: The device plays audio data and communicates a response to the user. For example, it might play the audio "What are its features?".

[1305] Step 4: Persona-based training

[1306] Input: Persona information selected by the user

[1307] Operation: The user selects a specific customer type or persona using the device's persona selection interface. For example, they might select "Customers with complaints."

[1308] Data processing: The device sends the selected persona information to the server, and the server prepares dialogue patterns based on the persona information.

[1309] Output: Dialogue patterns based on the selected persona

[1310] Operation: The server generates a scenario based on the dialogue pattern and sends the initial question and situation description as text data to the terminal. The terminal converts the text data into speech and plays the scenario. For example, it plays a scenario such as, "This product arrived damaged, could I request a replacement?"

[1311] Step 5: Record and evaluate the content of the speech.

[1312] Input: User's utterance

[1313] Operation: The device records the user's speech in real time and sends it to the server.

[1314] Data processing: The server scores the user's utterances based on evaluation criteria. For example, it evaluates elements such as the content, timing, and politeness of the utterance.

[1315] Output: Evaluation results and points

[1316] Operation: The server calculates points based on the evaluation results and updates the points.

[1317] Step 6: Generating and providing feedback

[1318] Input: Evaluation results and points

[1319] Operation: The server generates a feedback message and sends it to the terminal to inform the user.

[1320] Data processing: The feedback message is generated as text data.

[1321] Output: Feedback message

[1322] Action: The device displays and provides a feedback message to the user. For example, the feedback might say, "The service you provided was excellent."

[1323] In this way, the system effectively trains users' customer service skills through a series of steps, and provides evaluation and feedback.

[1324] (Application Example 1)

[1325] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1326] Conventional customer service training systems use speech recognition and natural language processing technologies to interact with users, but they have the drawback of not being able to evaluate actions and speech during training in real time. Furthermore, they lacked training that took user actions into account and detailed feedback on speech content. As a result, there was a problem in that improvements in customer service skills in actual stores were not sufficiently achieved.

[1327] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1328] In this invention, the server includes means for converting the user's speech into text using a speech recognition engine; means for analyzing the text data using natural language processing technology to understand the user's intent; means for generating a response based on the intent and providing the generated response as text data; means for capturing the user's actions with a camera and evaluating the synchronized speech content and actions; means for generating a response based on the user's speech content using a generative AI model; means for providing the generated response to the user as evaluation and feedback; and means for storing the speech content and evaluation results in a database. This makes it possible to grasp the user's actions and speech content in real time and provide evaluation and feedback on the spot.

[1329] A "speech recognition engine" is a device or software that captures a user's speech as a digital signal and converts that speech into text data.

[1330] "Natural language processing technology" is a technology that analyzes text data, understands the user's intentions and emotions, and enables natural dialogue between humans and computers.

[1331] "Means for generating responses" refer to algorithms or systems that construct appropriate responses based on the user's intent and provide them as text or audio data.

[1332] A "server device" is a central processing unit that receives and processes data from multiple users and provides necessary information and services.

[1333] A "terminal device" is a device that a user directly operates, equipped with a microphone, speaker, display, etc., and functions in conjunction with a server.

[1334] A "camera" is a device that captures a user's actions and gestures in real time and transmits that information to a server as image data.

[1335] A "generative AI model" is an artificial intelligence model that learns from a large dataset and generates natural-sounding responses based on the user's utterances.

[1336] A "prompt sentence" is an input sentence used by a generative AI model to generate a dialogue response, and it is an instruction sentence that appropriately reflects the user's intent.

[1337] A "database" is an information management system that stores and manages information such as spoken content, evaluation results, and dialogue history, and allows for quick reference as needed.

[1338] "Feedback" refers to information provided in real time regarding the evaluation and areas for improvement of the user's speech and actions during training.

[1339] This invention relates to a customer service training system using smart glasses, which provides real-time feedback utilizing speech recognition, natural language processing, and generative AI models. The system consists of a server device, terminal devices, and a network connecting them. Specific embodiments for carrying out the invention are described below.

[1340] System Configuration

[1341] The system consists of the following elements:

[1342] 1. Server equipment:

[1343] Speech recognition engine: Software that converts a user's speech into text data.

[1344] Natural Language Processing (NLP) technology: Software that analyzes text data to understand user intent.

[1345] Generative AI Model: This is an artificial intelligence model that generates responses based on the user's utterances. Specifically, it uses the Hugging Face transformers library and the GPT-3.5 model.

[1346] Database: A system that stores and manages spoken content, evaluation results, and dialogue history.

[1347] Response generation algorithm: An algorithm that generates an appropriate response based on the results of natural language processing.

[1348] Evaluation and Point Calculation System: A system that evaluates users in real time based on their speech and actions and calculates points.

[1349] 2. Terminal device (smart glasses):

[1350] Camera and microphone: Devices for capturing user actions and voice in real time.

[1351] Text-to-speech engine: Software that converts text data received from a server into speech and plays it back. Specifically, it uses the gTTS library.

[1352] Feedback display interface: An interface for displaying evaluation results and areas for improvement to the user.

[1353] Persona Selection Interface: An interface for users to select training scenarios.

[1354] Operation of the invention

[1355] 1. Customer service training begins:

[1356] The user wears smart glasses and selects a training scenario using a persona selection interface. This sends persona information to the server.

[1357] 2. Speech recognition and motion capture:

[1358] The smart glasses' microphone records the user's voice, and the camera captures their movements. The recorded audio data is sent to a server and converted into text data by a speech recognition engine.

[1359] 3. Natural Language Processing and Response Generation:

[1360] The server analyzes the text data and uses a generative AI model to generate an appropriate response. The generated response is then sent to the terminal device as text data.

[1361] 4. Speech synthesis and playback:

[1362] The terminal device's speech synthesis engine converts text data into speech and plays it back to the user.

[1363] 5. Evaluation and Feedback:

[1364] The server evaluates the user's speech and actions in real time and generates feedback. The evaluation results are stored in a database, and the feedback is provided to the user through a terminal device.

[1365] Specific example

[1366] As a concrete example, let's explain the introduction of a new product. When a user introduces a new product, the system operates in the following order:

[1367] 1. Scenario Selection:

[1368] The user selects "Introduce new products".

[1369] 2. Audio and motion capture:

[1370] The smart glasses capture the user's voice and movements, and the server analyzes them.

[1371] 3. Response generation:

[1372] User: "This product is a convenient product that uses the latest technology."

[1373] Server: The generation AI model generates a response based on the prompt text and responds with "What specific technologies are being used?".

[1374] 4. Audio Playback:

[1375] The smart glasses play the response aloud.

[1376] 5. Evaluation and Feedback:

[1377] The server evaluates the user's speech in real time and provides feedback such as, "It would be good if you could explain the product's features in more detail."

[1378] Example of a prompt

[1379] "The user is describing a new product. Please generate a natural-sounding continuation of the conversation. User: 'This product is a convenient product that uses the latest technology.' Please generate the AI's subsequent response."

[1380] This invention allows staff working in physical stores to efficiently improve their customer service skills while receiving real-time evaluation of user actions and speech. Furthermore, the use of a generative AI model enhances the naturalness and diversity of the dialogue.

[1381] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1382] Step 1:

[1383] The server receives information about the training scenario selected by the user (e.g., "Introduction to a New Product") from the persona selection interface of the terminal device (smart glasses). The input data is the persona information selected by the user, and the server prepares dialogue patterns corresponding to that persona as output.

[1384] Step 2:

[1385] The device uses a microphone built into the smart glasses to record the user's speech in real time. The input data is the voice spoken by the user, and the output data is the digital signal of that voice. This digital signal is sent to a server.

[1386] Step 3:

[1387] The server analyzes the received audio data using a speech recognition engine and converts it into text data. The input data is a digital audio signal, and the output data is the text data obtained by analyzing that audio signal.

[1388] Step 4:

[1389] The server analyzes text data using natural language processing techniques to understand the user's intent. The input data is text data obtained by a speech recognition engine, and the output data is metadata containing the user's intent as a result of the analysis.

[1390] Step 5:

[1391] The server uses a generative AI model based on the user's intent to generate an appropriate response. The input data consists of metadata and prompts for the generative AI model, parsed using natural language processing techniques, while the output data is the text data of the generated response.

[1392] Step 6:

[1393] The server sends the generated response text data to the terminal device. The input data is the response text data, and the output is the transmission of the text data to the terminal device.

[1394] Step 7:

[1395] The terminal converts the received response text data into speech using a speech synthesis engine and plays it back to the user. The input data is text data received from the server, and the output data is speech generated by speech synthesis.

[1396] Step 8:

[1397] The device uses the smart glasses' camera to capture the user's movements and sends the video data to the server. The input data is the video captured by the camera, and the output data is the digital signal of that video.

[1398] Step 9:

[1399] The server evaluates the user's speech and motion data and generates feedback. The input data consists of audio and video data transmitted from the smart glasses, and the output data is a feedback message representing the evaluation result.

[1400] Step 10:

[1401] The server sends the generated feedback message to the terminal device and plays it back to the user either visually or audibly. The input data is the feedback message, and the output data is the transmission of the message to the terminal device and its display or audibility.

[1402] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1403] This invention relates to a customer service training system that combines voice dialogue AI technology and emotion recognition. This system recognizes the user's voice and understands the user's intentions based on natural language processing technology, as well as recognizing the user's emotions using an emotion engine. Based on this information, it generates appropriate responses to improve skills. Specific embodiments of this invention are described below.

[1404] System Configuration

[1405] 1. Server equipment

[1406] Speech recognition engine

[1407] Natural Language Processing Engine

[1408] Emotional Engine

[1409] Response generation algorithm

[1410] Evaluation and point calculation system

[1411] Database (user data, persona information, dialogue patterns, sentiment data)

[1412] 2. Terminal device

[1413] Microphone and speaker

[1414] Speech synthesis engine

[1415] Point and feedback display interface

[1416] Persona Selection Interface

[1417] Specific steps for implementing the invention

[1418] Users of this system use a terminal device to begin customer service training, including emotion recognition. When a user starts a scenario, the following processes occur:

[1419] 1. Voice recording and recognition

[1420] The user presses the recording start button on the terminal device.

[1421] The (device) uses the microphone to record the (user's) voice.

[1422] The (device) sends the recorded audio data to the (server).

[1423] The server uses a speech recognition engine to convert the speech data into text data.

[1424] 2. Natural Language Processing and Response Generation

[1425] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent.

[1426] The server generates an appropriate response based on the intent and sends that response as text data to the terminal.

[1427] The (device) converts the text data into speech using a speech synthesis engine and plays the response back to the (user).

[1428] 3. Emotion recognition

[1429] The server passes voice and text data to the emotion engine, which analyzes the user's emotions.

[1430] The server adjusts its response based on emotional data to generate more appropriate dialogue.

[1431] 4. Persona-based training

[1432] The user selects a persona on the terminal device (e.g., "Customer with a complaint").

[1433] The (device) sends the selected persona information to the (server).

[1434] The server references persona-based dialogue patterns and prepares the scenario.

[1435] The server generates the first question according to the scenario and sends it to the terminal as text data.

[1436] The (device) converts the text data into speech and plays it back to the user.

[1437] 5. Evaluation and Point System

[1438] The (terminal) records the user's speech in real time and sends it to the (server).

[1439] The server scores the speech content and emotional data based on evaluation criteria.

[1440] The server calculates points based on the evaluation results and updates the user's performance database.

[1441] The server generates a feedback message and sends it to the terminal.

[1442] The (device) displays and provides a feedback message to the user.

[1443] Specific example

[1444] Example 1: Product Description Practice

[1445] The user selects a persona for the "product description" and starts the scenario.

[1446] The (device) started recording, and the AI ​​bot said, "Could you please explain this new product?"

[1447] The user says, "This product uses the latest technology..."

[1448] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[1449] The server recognizes the user's emotions from their voice and generates feedback that corresponds to those emotions.

[1450] The (device) plays back questions and feedback via audio, continuing the conversation.

[1451] Example 2: Practicing handling complaints

[1452] The user selects the "Complaint Handling" persona and starts the scenario.

[1453] The (device) started recording, and the AI ​​bot said, "This product arrived damaged, so could I request an exchange?"

[1454] The user says, "I am very sorry, I will take care of it immediately."

[1455] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[1456] The server uses an emotion engine to recognize the user's anger state and generates an additional response to calm them down.

[1457] The (device) plays the question and additional responses aloud, continuing the conversation.

[1458] This system is designed to allow users to efficiently acquire the skills necessary to handle a wide range of scenarios while engaging in actual conversations. Furthermore, the emotion recognition function enables the creation of even more realistic conversation scenarios.

[1459] The following describes the processing flow.

[1460] Utilization of voice-activated AI bots and emotion recognition

[1461] Step 1:

[1462] The user presses the recording start button on the terminal device.

[1463] The (device) uses the microphone to record the user's voice.

[1464] Step 2:

[1465] The (device) saves the recorded audio data to a buffer.

[1466] The (terminal) sends the audio data in the buffer to the (server).

[1467] Step 3:

[1468] The server starts the speech recognition engine and converts the received speech data into text data.

[1469] The server generates the text data of the conversion result.

[1470] Step 4:

[1471] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[1472] The server generates a response based on the intended meaning.

[1473] Step 5:

[1474] The server passes voice and text data to the emotion engine, which analyzes the user's emotions.

[1475] The server adjusts its response based on the emotion recognition results.

[1476] Step 6:

[1477] The server generates a pre-arranged response as text data and sends it to the terminal.

[1478] The text data received by the terminal is converted into speech using a speech synthesis engine.

[1479] The (device) plays the converted audio through the speaker.

[1480] Persona-based training

[1481] Step 1:

[1482] The user uses the terminal device's interface to select a specific customer type or persona.

[1483] The (device) sends the selected persona information to the (server).

[1484] Step 2:

[1485] The server processes the persona information it receives.

[1486] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[1487] Step 3:

[1488] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[1489] The (device) receives text data, converts it into speech, and plays it back to the (user).

[1490] Role-playing progress evaluation and point system

[1491] Step 1:

[1492] The (device) records the user's speech in real time.

[1493] The (terminal) sends the recorded speech data to the (server).

[1494] Step 2:

[1495] The server analyzes the received speech data and evaluates its appropriateness and timing based on pre-set evaluation criteria.

[1496] The server calculates a score for each evaluation item based on speech data and emotion recognition data.

[1497] Step 3:

[1498] The server calculates points based on the evaluation results.

[1499] The server adds points and updates the user's performance data.

[1500] Step 4:

[1501] The server sends the generated points and performance information to the terminal.

[1502] The terminal displays the points and performance information it has received and provides it to the user.

[1503] Feedback and improvement loop

[1504] Step 1:

[1505] The server re-evaluates the user's utterance and generates specific feedback.

[1506] The server sends the generated feedback message to the terminal as text data.

[1507] Step 2:

[1508] The (device) displays the feedback message it received on the screen.

[1509] The user reviews the feedback message and understands the areas for improvement.

[1510] (Example 2)

[1511] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1512] Conventional customer service training systems could understand user intent and generate responses using speech recognition and natural language processing technologies, but they struggled to respond while considering user emotions. Furthermore, there were few means to evaluate the effectiveness of the training, making it difficult to efficiently improve user performance and skills. Additionally, training based on specific scenarios was limited, making it difficult to simulate real-world customer service situations.

[1513] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1514] In this invention, the server includes means for converting user speech to text using a speech recognition engine, means for analyzing text data using natural language processing technology to understand user intent, means for generating a response based on intent and providing the generated response as text data, means for converting text data to speech and playing it back to the user, means for analyzing speech and text data to recognize user emotions, means for adjusting responses based on emotions to generate more appropriate dialogue, means for providing an interface for selecting persona information, means for preparing dialogue patterns based on the selected persona, means for starting a scenario based on the dialogue pattern and generating a speech response corresponding to the scenario, means for recording and evaluating user utterances, means for calculating points based on evaluation and updating points, and means for displaying points and performance. This enables the provision of appropriate responses that take emotions into account, improves the quality of dialogue, and allows for evaluation of user skills and confirmation of performance. Furthermore, by realizing training based on specific scenarios, it becomes possible to simulate situations that closely resemble actual customer service scenarios.

[1515] A "speech recognition engine" is a technology that takes speech data as input and converts it into text data.

[1516] "Natural language processing" is a technology that analyzes text data, understands human language, and generates appropriate responses.

[1517] A "server device" is a computer system that provides data processing and storage functions and operates in conjunction with client devices.

[1518] A "terminal device" is a device that a user directly operates and that provides interface functions while communicating with a server.

[1519] An "emotion engine" is a technology that analyzes voice and text data to estimate a user's emotional state.

[1520] A "persona" is a fictional character based on a specific scenario or user group, and is the subject of simulation in a training system.

[1521] A "dialogue pattern" is a definition of the expected sequence of questions and answers in a particular scenario.

[1522] "Evaluation criteria" are standards or standards used to evaluate user performance based on the content of user utterances and emotional state.

[1523] "Feedback" refers to evaluation comments and advice provided regarding a user's performance.

[1524] System Overview

[1525] This invention relates to a customer service training system that combines voice dialogue AI technology and emotion recognition. This system recognizes the user's voice and understands their intentions based on natural language processing technology, as well as recognizing the user's emotions using an emotion engine. Based on this information, it generates appropriate responses to improve skills.

[1526] Hardware and software configuration

[1527] This system consists of the following components:

[1528] Server equipment

[1529] Speech recognition engine: Converts speech data into text (e.g., Google Cloud Speech-to-Text)

[1530] Natural language processing engine: Analyzes text data and understands user intent (e.g., GPT-3).

[1531] Emotion engine: Analyzes user emotions based on voice and text data (e.g., Microsoft Azure Emotion API)

[1532] Response generation algorithm: Generates appropriate responses based on the user's intent and emotions.

[1533] Evaluation and point calculation system: Evaluates the user's utterances and calculates points.

[1534] Database: Stores user data, persona information, dialogue patterns, and sentiment data.

[1535] Terminal device

[1536] Microphone and speaker: Used for audio input and output.

[1537] Speech synthesis engine: Converts text data into speech (e.g., Amazon Polly)

[1538] Point and feedback display interface: Displays points and feedback to the user.

[1539] Persona Selection Interface: An interface for users to select a persona.

[1540] Specific implementations of the system

[1541] Users of this system use a terminal device to begin customer service training, including emotion recognition. Specific embodiments of the present invention are shown below.

[1542] Voice recording and recognition

[1543] The user presses the recording start button on the terminal device.

[1544] The device uses the microphone to record the user's voice.

[1545] The device sends the recorded audio data to the server.

[1546] The server uses a speech recognition engine to convert the audio data into text data.

[1547] Natural language processing and response generation

[1548] The server analyzes the text data using a natural language processing engine to understand the user's intent.

[1549] The server generates an appropriate response based on the user's intent and sends that response as text data to the terminal.

[1550] The device converts text data into speech using a speech synthesis engine and plays the response back to the user.

[1551] emotion recognition

[1552] The server passes voice and text data to the emotion engine, which then analyzes the user's emotions.

[1553] The server adjusts its response based on emotional data to generate more appropriate dialogue.

[1554] Persona-based training

[1555] The user selects a persona (e.g., "a customer with a complaint") on the terminal device.

[1556] The device sends the selected persona information to the server.

[1557] The server references persona-based dialogue patterns and prepares the scenario.

[1558] The server generates an initial question based on the scenario and sends it to the terminal as text data.

[1559] The device converts text data into speech and plays it back to the user.

[1560] Rating and point system

[1561] The device records the user's speech in real time and sends it to the server.

[1562] The server scores the spoken content and emotional data based on evaluation criteria.

[1563] The server calculates points based on the evaluation results and updates the user's performance database.

[1564] The server generates a feedback message and sends it to the terminal.

[1565] The device displays and provides a feedback message to the user.

[1566] Specific usage examples

[1567] Product description practice

[1568] The user selects a persona for the "product description" and starts the scenario.

[1569] The device starts recording and says, "Could you please explain this new product?"

[1570] The user says, "This product uses the latest technology..."

[1571] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[1572] The server recognizes emotions from the user's voice and generates emotionally appropriate feedback.

[1573] The device plays back questions and feedback via voice, continuing the conversation.

[1574] Practice in handling complaints

[1575] The user selects the "complaint handling" persona and starts the scenario.

[1576] The device starts recording and says, "This product arrived damaged, so could I request a replacement?"

[1577] The user says, "I am very sorry, I will take care of it immediately."

[1578] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[1579] The server uses an emotion engine to recognize the user's anger level and generates additional responses to calm them down.

[1580] The device plays back the question and any additional responses via audio, continuing the conversation.

[1581] Examples of prompts for generative AI models

[1582] "Please provide a detailed explanation of the new product."

[1583] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1584] Step 1:

[1585] Audio recording and transfer

[1586] The user presses the recording start button on the terminal device. The input is the user's voice.

[1587] The device uses the microphone to record the user's voice. The output is the recorded audio data.

[1588] The device sends the recorded audio data to the server. The audio data is transferred from the device to the server.

[1589] Step 2:

[1590] Text conversion of audio data

[1591] The server uses a speech recognition engine to convert audio data into text data. The input is recorded audio data.

[1592] A speech recognition engine (such as Google Cloud Speech-to-Text) analyzes the audio data and converts it into text data. The output is text data.

[1593] Step 3:

[1594] Natural language processing and intent analysis

[1595] The server analyzes text data using a natural language processing engine. The input is text data.

[1596] A natural language processing engine (such as GPT-3) analyzes text data to understand the user's intent. The output is data that reflects the user's intent.

[1597] Step 4:

[1598] Response generation

[1599] The server generates an appropriate response based on the user's intent. The input is parsed intent data.

[1600] The response generation algorithm creates a text response tailored to the user's intent. The output is the text data of the response.

[1601] Step 5:

[1602] Speech and playback of responses

[1603] The server sends the generated text data to the terminal. The input is the response text data.

[1604] The device converts text data into speech using a speech synthesis engine (such as Amazon Polly). The output is the generated speech data.

[1605] The terminal plays the generated audio to the user. The audio data is provided to the user through the audio output device.

[1606] Step 6:

[1607] emotion recognition

[1608] The server passes audio and text data to the emotion engine. The input consists of audio and text data.

[1609] An emotion engine (such as the Microsoft Azure Emotion API) analyzes the data and recognizes the user's emotions. The output is emotion data.

[1610] The server adjusts its response based on sentiment data and generates a more appropriate dialogue. Adjusted response data is then generated.

[1611] Step 7:

[1612] Persona-based training begins.

[1613] The user selects a persona (e.g., "Customer with a complaint") on the terminal device. Persona information is then entered.

[1614] The device sends the selected persona information to the server. The input is the selected persona data.

[1615] The server references persona-based dialogue patterns and prepares a scenario. The output is scenario data.

[1616] The server generates the first question based on the scenario and sends it to the terminal as text data. The input is the scenario data.

[1617] The device converts text data into speech and plays it back to the user. The output is audio data.

[1618] Step 8:

[1619] Recording and evaluation of spoken content

[1620] The terminal records the user's speech in real time and sends it to the server. The input is the user's speech.

[1621] The server scores the utterance content and emotional data based on evaluation criteria. The input consists of the utterance content and emotional data.

[1622] The server calculates points based on the evaluation results and updates the user's performance database. The output is the updated performance data.

[1623] Step 9:

[1624] Generating and displaying feedback

[1625] The server generates a feedback message and sends it to the terminal. The input is the evaluation result.

[1626] The device displays and provides a feedback message to the user. The output is the feedback message.

[1627] (Application Example 2)

[1628] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1629] Conventional customer service training systems only utilize dialogue based on speech recognition and natural language processing technologies, and are unable to recognize user emotions and provide appropriate responses and feedback based on them. Therefore, it was difficult to effectively train staff in the ability to understand user emotions and respond appropriately in real-world customer service situations. Furthermore, the inability to conduct real-time training using cutting-edge technologies such as smart glasses meant there was a lack of means to improve training efficiency.

[1630] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting the user's voice into text using a speech recognition engine, means for analyzing the text data using natural language processing technology to understand the user's intent, and means for providing emotional recognition engines that analyze the user's emotions and provide emotionally appropriate responses and feedback messages. This makes it possible to recognize the user's voice in real time, generate responses based on their intent and emotions, and perform training through smart glasses.

[1631] A "speech recognition engine" is a technology used to convert speech data into text data.

[1632] "Natural language processing technology" is an information processing technology that analyzes text data to understand the user's intent.

[1633] An "emotion recognition engine" is a technology that analyzes a user's emotions based on voice data and text data.

[1634] A "server device" is a computer system that performs various functions such as data processing and analysis, response generation, and feedback provision.

[1635] A "terminal device" is a user device that communicates with a server device and performs functions such as audio recording and playback, and interface display.

[1636] "Text data" refers to character information converted by a speech recognition engine.

[1637] A "response generation algorithm" is a program that creates an appropriate response based on the user's intentions and emotions.

[1638] "Smart glasses" are devices that, when worn by the user, provide visual information and interactive interfaces.

[1639] A "feedback message" is a message generated based on the user's training results and emotions, intended for evaluation and improvement.

[1640] A "persona" is a virtual character or role used to simulate a specific dialogue scenario or situation.

[1641] This invention is a system that combines voice dialogue AI technology and emotion recognition technology to provide real-time customer service training through smart glasses. The following describes embodiments for carrying out this invention.

[1642] System Configuration

[1643] 1. Server equipment

[1644] Speech recognition engine

[1645] Natural Language Processing Technology

[1646] Emotion recognition engine

[1647] Response generation algorithm

[1648] Evaluation and point calculation system

[1649] Database (user data, persona information, dialogue patterns, sentiment data)

[1650] 2. Terminal device

[1651] Smart Glasses

[1652] Microphone and speaker

[1653] Speech synthesis engine

[1654] Point and feedback display interface

[1655] Persona Selection Interface

[1656] Program details

[1657] The server device performs the following steps when a user uses a terminal device (smart glasses) to conduct customer service training.

[1658] 1. Voice recording and recognition

[1659] The server device receives the user's voice data collected by the terminal device and converts it into text data using a speech recognition engine.

[1660] 2. Natural Language Processing and Response Generation

[1661] The server device analyzes the converted text data using natural language processing technology to understand the user's intent. Next, it generates an appropriate response based on that intent using a response generation algorithm and sends it to the terminal device as text data.

[1662] 3. Emotion recognition and adjustment of responses

[1663] The server device passes voice and text data to the emotion recognition engine, which analyzes the user's emotions. Based on the analyzed emotion data, it adjusts its response and generates a more appropriate feedback message.

[1664] 4. Real-time training via smart glasses

[1665] The smart glasses, which are the terminal device, convert the received response into speech using a speech synthesis engine and play it back to the user. They also visually display the evaluation results to the user through a points and feedback display interface.

[1666] Devices and Software

[1667] The server device has a program installed to analyze the collected data and generate appropriate responses. The hardware and software used are as follows:

[1668] Speech recognition engine: Google Speech Recognition API

[1669] Natural Language Processing Techniques: Hugging Face's Transformers Library

[1670] Emotion Recognition Engine: Emotion Analysis Pipeline using the Transformers Library

[1671] Speech synthesis engine: Google Speech API

[1672] Smart glasses: Devices that provide visual and auditory information when worn by the user.

[1673] Specific example

[1674] When a store clerk speaks through smart glasses and says, "Could you tell me more about this product?", the system converts the speech into text and analyzes the clerk's emotions. If the clerk is excited, the system responds, "That's a great suggestion! Do you have any further questions?", and if the clerk is confused, it responds, "I'm sorry. I'll do my best to resolve the issue." The system always provides guidance that is appropriate to the clerk's emotions.

[1675] Example of a prompt:

[1676] User's utterance:

[1677] "Please tell me more about this product."

[1678] Example system response:

[1679] "That's a great suggestion! Do you have any further questions?"

[1680] User's emotional state: positive

[1681] Expected system response:

[1682] "That's a great suggestion! Do you have any further questions?"

[1683] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1684] Step 1:

[1685] The user puts on the smart glasses and starts training mode. The user selects the persona they want to train (e.g., "product description" or "customer complaint handling") through the persona selection interface. The device sends the selected persona information to the server.

[1686] Step 2:

[1687] The server retrieves appropriate dialogue patterns from the database based on the selected persona and prepares a training scenario. The server generates an initial question or situation description and sends the text data to the terminal.

[1688] Step 3:

[1689] The device converts the received text data into speech using a speech synthesis engine and plays it back to the user. The user then inputs the next action or response via voice.

[1690] Step 4:

[1691] The terminal records the user's voice input and sends it to the server as audio data. The server uses a speech recognition engine to convert the audio data into text data. The voice input is a specific question, such as "Please tell me more about this product."

[1692] Step 5:

[1693] The server analyzes the converted text data using natural language processing techniques to understand the user's intent, such as a specific request like "Please tell me the details of the product." Based on this analysis, the server generates the next appropriate response using a response generation algorithm.

[1694] Step 6:

[1695] The server analyzes the user's voice and text data using an emotion recognition engine to identify the user's emotional state. For example, it can determine whether the user is excited or confused.

[1696] Step 7:

[1697] The server adjusts its response based on the emotions identified by the emotion recognition engine, generating a more appropriate feedback message. For example, if the user is confused, it might respond with, "We're sorry. We'll do our best to resolve the issue." This response is then sent to the terminal as text data.

[1698] Step 8:

[1699] The device converts the received text data back into speech using a speech synthesis engine and plays it back to the user. This allows the user to receive feedback in real time.

[1700] Step 9:

[1701] The server passes the entire conversation record to the evaluation system, which scores the user's performance. Based on the evaluation results, points are calculated and the user's performance database is updated.

[1702] Step 10:

[1703] The device displays points and evaluation results to the user through a feedback display interface. For example, it provides evaluations using specific numerical values, such as "You earned 10 points in today's training."

[1704] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1705] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1706] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1707] [Fourth Embodiment]

[1708] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1709] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1710] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1711] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1712] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1713] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1714] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1715] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1716] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1717] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1718] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1719] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1720] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1721] This invention relates to a customer service training system that utilizes voice dialogue AI technology. This system aims to improve users' customer service skills through natural conversation using speech recognition and natural language processing (NLP). Specific embodiments of this invention are described below.

[1722] System Configuration

[1723] 1. Server equipment

[1724] Speech recognition engine

[1725] Natural Language Processing Engine

[1726] Response generation algorithm

[1727] Evaluation and point calculation system

[1728] Database (user data, persona information, dialogue patterns)

[1729] 2. Terminal device

[1730] Microphone and speaker

[1731] Speech synthesis engine

[1732] Point and feedback display interface

[1733] Persona Selection Interface

[1734] Specific steps for implementing the invention

[1735] Users of this system use a terminal device to begin customer service training. When a user starts a customer service scenario, the following processes occur.

[1736] 1. Voice recording and recognition

[1737] The (device) displays a recording start button, and the (user) presses the recording start button.

[1738] The (device) uses the microphone to record the (user's) voice.

[1739] The (device) sends the recorded audio data to the (server).

[1740] The server uses a speech recognition engine to convert the speech data into text data.

[1741] 2. Natural Language Processing and Response Generation

[1742] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[1743] The server generates an appropriate response based on the intent and sends that response as text data to the terminal.

[1744] The (device) converts text data into speech using a speech synthesis engine and plays the response back to the (user).

[1745] 3. Persona-based training

[1746] The user uses the terminal device's persona selection interface to select a specific customer type or persona (e.g., "Customer with a complaint").

[1747] The (device) sends the selected persona information to the (server).

[1748] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[1749] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[1750] The (device) converts the text data into speech and plays it back to the user.

[1751] As the training progresses, various persona-based dialogue scenarios unfold seamlessly.

[1752] 4. Evaluation and Feedback

[1753] The (terminal) records the user's speech in real time and sends it to the (server).

[1754] The server scores the user's utterances based on evaluation criteria.

[1755] The server calculates points based on the evaluation results and updates the user's performance database.

[1756] The server generates a feedback message and sends it to the terminal.

[1757] The (device) displays and provides a feedback message to the user.

[1758] Specific example

[1759] Example 1: Product Description Practice

[1760] The user selects a persona for the "product description" and starts the scenario.

[1761] The (device) started recording, and the AI ​​bot said, "Could you please explain this new product?"

[1762] The user says, "This product uses the latest technology..."

[1763] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[1764] The (device) plays the question aloud and continues the conversation.

[1765] Example 2: Practicing handling complaints

[1766] The user selects the "Complaint Handling" persona and starts the scenario.

[1767] The (device) started recording, and the AI ​​bot said, "This product arrived damaged, so could I request an exchange?"

[1768] The user says, "I am very sorry, I will take care of it immediately."

[1769] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[1770] The (device) plays the question aloud and continues the conversation.

[1771] Thus, this system is designed to allow users to efficiently acquire the skills necessary to handle a wide range of scenarios while actually engaging in dialogue.

[1772] The following describes the processing flow.

[1773] Utilization of voice-activated AI bots

[1774] Step 1:

[1775] The user presses the recording start button on the terminal device.

[1776] The (device) uses the microphone to record the user's voice.

[1777] Step 2:

[1778] The (device) saves the recorded audio data to a buffer.

[1779] The (terminal) sends the audio data in the buffer to the (server).

[1780] Step 3:

[1781] The server starts the speech recognition engine and converts the received speech data into text data.

[1782] The server generates the text data of the conversion result.

[1783] Step 4:

[1784] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[1785] The server generates a response based on the intended meaning.

[1786] Step 5:

[1787] The server generates a response as text data and sends it to the terminal.

[1788] The text data received by the terminal is converted into speech using a speech synthesis engine.

[1789] The (device) plays the converted audio through the speaker.

[1790] Persona-based training

[1791] Step 1:

[1792] The user uses the terminal device's interface to select a specific customer type or persona.

[1793] The (device) sends the selected persona information to the (server).

[1794] Step 2:

[1795] The server processes the persona information it receives.

[1796] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[1797] Step 3:

[1798] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[1799] The (device) receives text data, converts it into speech, and plays it back to the (user).

[1800] Role-playing progress evaluation and point system

[1801] Step 1:

[1802] The (device) records the user's speech in real time.

[1803] The (terminal) sends the recorded speech data to the (server).

[1804] Step 2:

[1805] The server analyzes the received speech data and evaluates its appropriateness and timing based on pre-set evaluation criteria.

[1806] The server calculates a score for each evaluation item.

[1807] Step 3:

[1808] The server calculates points based on the evaluation results.

[1809] The server adds points and updates the user's performance data.

[1810] Step 4:

[1811] The server sends the generated points and performance information to the terminal.

[1812] The terminal displays the points and performance information it has received and provides it to the user.

[1813] Feedback and improvement loop

[1814] Step 1:

[1815] The server re-evaluates the user's utterances and generates specific feedback.

[1816] The server sends the generated feedback message to the terminal as text data.

[1817] Step 2:

[1818] The (device) displays the feedback message it received on the screen.

[1819] The user reviews the feedback message and understands the areas for improvement.

[1820] (Example 1)

[1821] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1822] In customer service training, it is crucial for users to receive practical training based on diverse scenarios in order to effectively improve their conversational skills. However, conventional systems have not effectively performed real-time speech recognition or natural language processing, making it difficult to accurately understand and respond to user intentions. Furthermore, methods for evaluating user speech and providing feedback were insufficient, making it difficult for users to objectively evaluate their own skills and continuously improve them. Therefore, a new system is needed to effectively improve users' customer service skills.

[1823] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1824] In this invention, the server includes speech recognition means for converting user speech into text data, natural language processing means for analyzing the text data and understanding the user's intent, and response generation means for generating a response based on the user's intent and providing the generated response as text data. This enables users to engage in real-time dialogue based on diverse scenarios and effectively improve their customer service skills. Furthermore, by including means for providing an interface for selecting persona information, means for preparing dialogue patterns based on the selected persona, and means for starting a scenario based on the dialogue pattern and generating a voice response corresponding to the scenario, users can receive practical training according to diverse scenarios. In addition, by including means for recording and evaluating the user's utterances, means for calculating and updating points based on the evaluation, and means for displaying points and performance, users can objectively evaluate their own skills and continuously improve them.

[1825] "Voice recognition means" refers to a device or technology that records a user's voice and converts it into text data.

[1826] "Natural language processing" refers to technologies that analyze text data and understand the user's intent.

[1827] "Response generation means" refers to a device or technology that generates an appropriate response based on the user's intent and provides the generated response as text data.

[1828] "Speech synthesis means" refers to a device or technology that converts text data into speech and plays it back to the user.

[1829] "Persona information" refers to character information selected by specific customer types or users.

[1830] A "dialogue pattern" is a set of dialogue sequences and response patterns prepared based on a specific scenario.

[1831] A "scenario" is a sequence of dialogues based on a specific situation or case.

[1832] "Means for recording user speech content" refers to a device or technology that records the voice spoken by a user and stores it as data.

[1833] "Evaluation criteria" refer to the standards and rules used to evaluate the content of a user's utterances, and scoring is performed based on these criteria.

[1834] "Means for calculating points" refers to a device or technology that quantifies the user's utterances based on evaluation criteria and calculates them as points.

[1835] "Means for updating points" refers to a device or technology that reflects the calculated points in the user's performance database and updates it to the latest state.

[1836] An "interface" is a user interface or input device that allows a user to interact with a system.

[1837] A "feedback tool" is a device or technology that provides users with evaluation results, areas for improvement, and other information to help them improve their own skills.

[1838] This invention is a customer service training system that utilizes a speech recognition engine and natural language processing (NLP) technology. The system includes three main components: a server, a terminal, and a user.

[1839] First, the server includes a speech recognition engine, a natural language processing engine, a response generation algorithm, an evaluation and point calculation system, and a database that stores user data, persona information, and dialogue patterns. For the speech recognition engine, for example, the Google Cloud Speech-to-Text API can be used. For natural language processing, technologies such as spaCy or BERT are recommended. The response generation algorithm generates appropriate responses based on the user's intent and provides them as text data. The server also performs evaluations based on the user's utterances, calculates points, and generates feedback.

[1840] Next, the device includes a microphone and speaker, a speech synthesis engine, a point and feedback display interface, and a persona selection interface. For the speech synthesis engine, for example, Amazon Polly or Google Text-to-Speech can be used. The device records the user's voice and sends it to the server in real time. The text data received from the server is converted into speech and played back to the user. Furthermore, the persona selection interface allows the user to select specific customer types or personas.

[1841] The user receives customer service training using a device. For example, if the user selects the "product description" persona, the device starts recording and asks a voice question such as, "Could you please explain this new product?" When the user says, "This product uses the latest technology...", the device records the voice and sends it to the server. The server uses a speech recognition engine to convert the voice to text, analyzes it with a natural language processing engine, and then generates an appropriate response, "Could you tell me how to use it specifically?", and sends it to the device. The device converts that text data back into voice and plays it back to the user.

[1842] Furthermore, user performance is evaluated and calculated as points. The evaluation criteria are based on factors such as the content, timing, and politeness of the speech. Along with the evaluation results, points and feedback are provided to the user. For example, feedback such as "Your response this time was excellent" might be displayed.

[1843] Example of a prompt:

[1844] "Select a persona for your product description and begin explaining the new product."

[1845] "Let's select a persona for handling complaints and ask them to exchange the product."

[1846] In this way, users can efficiently improve their practical customer service skills through real-time interactions.

[1847] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1848] Step 1: Record and recognize audio

[1849] Input: User's speech

[1850] Operation: The device displays a recording start button, and recording begins when the user presses it. For example, the user might say, "Please explain the new product."

[1851] Data processing: The device's microphone records the user's voice and generates audio data.

[1852] Output: Recorded audio data

[1853] Operation: The device sends the recorded data to the server, and the server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio data into text data.

[1854] Step 2: Natural Language Processing and Response Generation

[1855] Input: Text data (converted from user's speech)

[1856] Operation: The server uses a natural language processing (NLP) engine (e.g., spaCy or BERT) to analyze text data and understand the user's intent. For example, it might analyze a request like "Please explain the new product."

[1857] Data processing: The NLP engine analyzes text data to identify the user's intent.

[1858] Output: User intent (e.g., the user is seeking a product description)

[1859] Operation: The server's response generation algorithm generates an appropriate response based on the user's intent and sends the generated response as text data to the terminal. For example, a response such as "What are its features?" might be generated.

[1860] Step 3: Speech synthesis and response playback

[1861] Input: Text data of the response

[1862] Operation: The device converts the received text data into speech using a text-to-speech engine (e.g., Amazon Polly or Google Text-to-Speech).

[1863] Data processing: The speech synthesis engine converts text data into speech data.

[1864] Output: Audio data

[1865] Operation: The device plays audio data and communicates a response to the user. For example, it might play the audio "What are its features?".

[1866] Step 4: Persona-based training

[1867] Input: Persona information selected by the user

[1868] Operation: The user selects a specific customer type or persona using the device's persona selection interface. For example, they might select "Customers with complaints."

[1869] Data processing: The device sends the selected persona information to the server, and the server prepares dialogue patterns based on the persona information.

[1870] Output: Dialogue patterns based on the selected persona

[1871] Operation: The server generates a scenario based on the dialogue pattern and sends the initial question and situation description as text data to the terminal. The terminal converts the text data into speech and plays the scenario. For example, it plays a scenario such as, "This product arrived damaged, could I request a replacement?"

[1872] Step 5: Record and evaluate the content of the speech.

[1873] Input: User's utterance

[1874] Operation: The device records the user's speech in real time and sends it to the server.

[1875] Data processing: The server scores the user's speech based on evaluation criteria. For example, it evaluates elements such as the content of the speech, timing, and politeness.

[1876] Output: Evaluation results and points

[1877] Operation: The server calculates points based on the evaluation results and updates the points.

[1878] Step 6: Generating and providing feedback

[1879] Input: Evaluation results and points

[1880] Operation: The server generates a feedback message and sends it to the terminal to inform the user.

[1881] Data processing: The feedback message is generated as text data.

[1882] Output: Feedback message

[1883] Action: The device displays and provides a feedback message to the user. For example, the feedback might say, "The service you provided was excellent."

[1884] In this way, the system effectively trains users' customer service skills through a series of steps, and provides evaluation and feedback.

[1885] (Application Example 1)

[1886] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1887] Conventional customer service training systems use speech recognition and natural language processing technologies to interact with users, but they have the drawback of not being able to evaluate actions and speech during training in real time. Furthermore, they lacked training that took user actions into account and detailed feedback on speech content. As a result, there was a problem in that improvements in customer service skills in actual stores were not sufficiently achieved.

[1888] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1889] In this invention, the server includes means for converting the user's speech into text using a speech recognition engine; means for analyzing the text data using natural language processing technology to understand the user's intent; means for generating a response based on the intent and providing the generated response as text data; means for capturing the user's actions with a camera and evaluating the synchronized speech content and actions; means for generating a response based on the user's speech content using a generative AI model; means for providing the generated response to the user as evaluation and feedback; and means for storing the speech content and evaluation results in a database. This makes it possible to grasp the user's actions and speech content in real time and provide evaluation and feedback on the spot.

[1890] A "speech recognition engine" is a device or software that captures a user's speech as a digital signal and converts that speech into text data.

[1891] "Natural language processing technology" is a technology that analyzes text data, understands the user's intentions and emotions, and enables natural dialogue between humans and computers.

[1892] "Means for generating responses" refer to algorithms or systems that construct appropriate responses based on the user's intent and provide them as text or audio data.

[1893] A "server device" is a central processing unit that receives and processes data from multiple users and provides necessary information and services.

[1894] A "terminal device" is a device that a user directly operates, equipped with a microphone, speaker, display, etc., and functions in conjunction with a server.

[1895] A "camera" is a device that captures a user's actions and gestures in real time and transmits that information to a server as image data.

[1896] A "generative AI model" is an artificial intelligence model that learns from a large dataset and generates natural-sounding responses based on the user's utterances.

[1897] A "prompt sentence" is an input sentence used by a generative AI model to generate a dialogue response, and it is an instruction sentence that appropriately reflects the user's intent.

[1898] A "database" is an information management system that stores and manages information such as spoken content, evaluation results, and dialogue history, and allows for quick reference as needed.

[1899] "Feedback" refers to information provided in real time regarding the evaluation and areas for improvement of the user's speech and actions during training.

[1900] This invention relates to a customer service training system using smart glasses, which provides real-time feedback utilizing speech recognition, natural language processing, and generative AI models. The system consists of a server device, terminal devices, and a network connecting them. Specific embodiments for carrying out the invention are described below.

[1901] System Configuration

[1902] The system consists of the following elements:

[1903] 1. Server equipment:

[1904] Speech recognition engine: Software that converts a user's speech into text data.

[1905] Natural Language Processing (NLP) technology: Software that analyzes text data to understand user intent.

[1906] Generative AI Model: This is an artificial intelligence model that generates responses based on the user's utterances. Specifically, it uses the Hugging Face transformers library and the GPT-3.5 model.

[1907] Database: A system that stores and manages spoken content, evaluation results, and dialogue history.

[1908] Response generation algorithm: An algorithm that generates an appropriate response based on the results of natural language processing.

[1909] Evaluation and Point Calculation System: A system that evaluates users in real time based on their speech and actions and calculates points.

[1910] 2. Terminal device (smart glasses):

[1911] Camera and microphone: Devices for capturing user actions and voice in real time.

[1912] Text-to-speech engine: Software that converts text data received from a server into speech and plays it back. Specifically, it uses the gTTS library.

[1913] Feedback display interface: An interface for displaying evaluation results and areas for improvement to the user.

[1914] Persona Selection Interface: An interface for users to select training scenarios.

[1915] Operation of the invention

[1916] 1. Customer service training begins:

[1917] The user wears smart glasses and selects a training scenario using a persona selection interface. This sends persona information to the server.

[1918] 2. Speech recognition and motion capture:

[1919] The smart glasses' microphone records the user's voice, and the camera captures their movements. The recorded audio data is sent to a server and converted into text data by a speech recognition engine.

[1920] 3. Natural Language Processing and Response Generation:

[1921] The server analyzes the text data and uses a generative AI model to generate an appropriate response. The generated response is then sent to the terminal device as text data.

[1922] 4. Speech synthesis and playback:

[1923] The terminal device's speech synthesis engine converts text data into speech and plays it back to the user.

[1924] 5. Evaluation and Feedback:

[1925] The server evaluates the user's speech and actions in real time and generates feedback. The evaluation results are stored in a database, and the feedback is provided to the user through a terminal device.

[1926] Specific example

[1927] As a concrete example, let's explain the introduction of a new product. When a user introduces a new product, the system operates in the following order:

[1928] 1. Scenario Selection:

[1929] The user selects "Introduce new products".

[1930] 2. Audio and motion capture:

[1931] The smart glasses capture the user's voice and movements, and the server analyzes them.

[1932] 3. Response generation:

[1933] User: "This product is a convenient product that uses the latest technology."

[1934] Server: The generation AI model generates a response based on the prompt text and responds with "What specific technologies are being used?".

[1935] 4. Audio Playback:

[1936] The smart glasses play the response aloud.

[1937] 5. Evaluation and Feedback:

[1938] The server evaluates the user's speech in real time and provides feedback such as, "It would be good if you could explain the product's features in more detail."

[1939] Example of a prompt

[1940] "The user is describing a new product. Please generate a natural-sounding continuation of the conversation. User: 'This product is a convenient product that uses the latest technology.' Please generate the AI's subsequent response."

[1941] This invention allows staff working in physical stores to efficiently improve their customer service skills while receiving real-time evaluation of user actions and speech. Furthermore, the use of a generative AI model enhances the naturalness and diversity of the dialogue.

[1942] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1943] Step 1:

[1944] The server receives information about the training scenario selected by the user (e.g., "Introduction to a New Product") from the persona selection interface of the terminal device (smart glasses). The input data is the persona information selected by the user, and the server prepares dialogue patterns corresponding to that persona as output.

[1945] Step 2:

[1946] The device uses a microphone built into the smart glasses to record the user's speech in real time. The input data is the voice spoken by the user, and the output data is the digital signal of that voice. This digital signal is sent to a server.

[1947] Step 3:

[1948] The server analyzes the received audio data using a speech recognition engine and converts it into text data. The input data is a digital audio signal, and the output data is the text data obtained by analyzing that audio signal.

[1949] Step 4:

[1950] The server analyzes text data using natural language processing techniques to understand the user's intent. The input data is text data obtained by a speech recognition engine, and the output data is metadata containing the user's intent as a result of the analysis.

[1951] Step 5:

[1952] The server uses a generative AI model based on the user's intent to generate an appropriate response. The input data consists of metadata and prompts for the generative AI model, parsed using natural language processing techniques, while the output data is the text data of the generated response.

[1953] Step 6:

[1954] The server sends the generated response text data to the terminal device. The input data is the response text data, and the output is the transmission of the text data to the terminal device.

[1955] Step 7:

[1956] The terminal converts the received response text data into speech using a speech synthesis engine and plays it back to the user. The input data is text data received from the server, and the output data is speech generated by speech synthesis.

[1957] Step 8:

[1958] The device uses the smart glasses' camera to capture the user's movements and sends the video data to the server. The input data is the video captured by the camera, and the output data is the digital signal of that video.

[1959] Step 9:

[1960] The server evaluates the user's speech and motion data and generates feedback. The input data consists of audio and video data transmitted from the smart glasses, and the output data is a feedback message representing the evaluation result.

[1961] Step 10:

[1962] The server sends the generated feedback message to the terminal device and plays it back to the user either visually or audibly. The input data is the feedback message, and the output data is the transmission of the message to the terminal device and its display or audibility.

[1963] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1964] This invention relates to a customer service training system that combines voice dialogue AI technology and emotion recognition. This system recognizes the user's voice and understands the user's intentions based on natural language processing technology, as well as recognizing the user's emotions using an emotion engine. Based on this information, it generates appropriate responses to improve skills. Specific embodiments of this invention are described below.

[1965] System Configuration

[1966] 1. Server equipment

[1967] Speech recognition engine

[1968] Natural Language Processing Engine

[1969] Emotional Engine

[1970] Response generation algorithm

[1971] Evaluation and point calculation system

[1972] Database (user data, persona information, dialogue patterns, sentiment data)

[1973] 2. Terminal device

[1974] Microphone and speaker

[1975] Speech synthesis engine

[1976] Point and feedback display interface

[1977] Persona Selection Interface

[1978] Specific steps for implementing the invention

[1979] Users of this system use a terminal device to begin customer service training, including emotion recognition. When a user starts a scenario, the following processes occur:

[1980] 1. Voice recording and recognition

[1981] The user presses the recording start button on the terminal device.

[1982] The (device) uses the microphone to record the (user's) voice.

[1983] The (device) sends the recorded audio data to the (server).

[1984] The server uses a speech recognition engine to convert the speech data into text data.

[1985] 2. Natural Language Processing and Response Generation

[1986] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent.

[1987] The server generates an appropriate response based on the intent and sends that response as text data to the terminal.

[1988] The (device) converts the text data into speech using a speech synthesis engine and plays the response back to the (user).

[1989] 3. Emotion recognition

[1990] The server passes voice and text data to the emotion engine, which analyzes the user's emotions.

[1991] The server adjusts its response based on emotional data to generate more appropriate dialogue.

[1992] 4. Persona-based training

[1993] The user selects a persona on the terminal device (e.g., "Customer with a complaint").

[1994] The (device) sends the selected persona information to the (server).

[1995] The server references persona-based dialogue patterns and prepares the scenario.

[1996] The server generates the first question according to the scenario and sends it to the terminal as text data.

[1997] The (device) converts the text data into speech and plays it back to the user.

[1998] 5. Evaluation and Point System

[1999] The (terminal) records the user's speech in real time and sends it to the (server).

[2000] The server scores the speech content and emotional data based on evaluation criteria.

[2001] The server calculates points based on the evaluation results and updates the user's performance database.

[2002] The server generates a feedback message and sends it to the terminal.

[2003] The (device) displays and provides a feedback message to the user.

[2004] Specific example

[2005] Example 1: Product Description Practice

[2006] The user selects a persona for the "product description" and starts the scenario.

[2007] The (device) started recording, and the AI ​​bot said, "Could you please explain this new product?"

[2008] The user says, "This product uses the latest technology..."

[2009] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[2010] The server recognizes the user's emotions from their voice and generates feedback that corresponds to those emotions.

[2011] The (device) plays back questions and feedback via audio, continuing the conversation.

[2012] Example 2: Practicing handling complaints

[2013] The user selects the "Complaint Handling" persona and starts the scenario.

[2014] The (device) started recording, and the AI ​​bot said, "This product arrived damaged, so could I request an exchange?"

[2015] The user says, "I am very sorry, I will take care of it immediately."

[2016] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[2017] The server uses an emotion engine to recognize the user's anger state and generates an additional response to calm them down.

[2018] The (device) plays the question and additional responses aloud, continuing the conversation.

[2019] This system is designed to allow users to efficiently acquire the skills necessary to handle a wide range of scenarios while engaging in actual conversations. Furthermore, the emotion recognition function enables the creation of even more realistic conversation scenarios.

[2020] The following describes the processing flow.

[2021] Utilization of voice-activated AI bots and emotion recognition

[2022] Step 1:

[2023] The user presses the recording start button on the terminal device.

[2024] The (device) uses the microphone to record the user's voice.

[2025] Step 2:

[2026] The (device) saves the recorded audio data to a buffer.

[2027] The (terminal) sends the audio data in the buffer to the (server).

[2028] Step 3:

[2029] The server starts the speech recognition engine and converts the received speech data into text data.

[2030] The server generates the text data of the conversion result.

[2031] Step 4:

[2032] The server uses a natural language processing (NLP) engine to analyze text data and understand the user's intent.

[2033] The server generates a response based on the intended meaning.

[2034] Step 5:

[2035] The server passes voice and text data to the emotion engine, which analyzes the user's emotions.

[2036] The server adjusts its response based on the emotion recognition results.

[2037] Step 6:

[2038] The server generates a pre-arranged response as text data and sends it to the terminal.

[2039] The text data received by the terminal is converted into speech using a speech synthesis engine.

[2040] The (device) plays the converted audio through the speaker.

[2041] Persona-based training

[2042] Step 1:

[2043] The user uses the terminal device's interface to select a specific customer type or persona.

[2044] The (device) sends the selected persona information to the (server).

[2045] Step 2:

[2046] The server processes the persona information it receives.

[2047] The server reads dialogue patterns based on the selected persona and prepares the scenario.

[2048] Step 3:

[2049] The server generates initial questions and situational descriptions based on the scenario and sends them to the terminal as text data.

[2050] The (device) receives text data, converts it into speech, and plays it back to the (user).

[2051] Role-playing progress evaluation and point system

[2052] Step 1:

[2053] The (device) records the user's speech in real time.

[2054] The (terminal) sends the recorded speech data to the (server).

[2055] Step 2:

[2056] The server analyzes the received speech data and evaluates its appropriateness and timing based on pre-set evaluation criteria.

[2057] The server calculates a score for each evaluation item based on speech data and emotion recognition data.

[2058] Step 3:

[2059] The server calculates points based on the evaluation results.

[2060] The server adds points and updates the user's performance data.

[2061] Step 4:

[2062] The server sends the generated points and performance information to the terminal.

[2063] The terminal displays the points and performance information it has received and provides it to the user.

[2064] Feedback and improvement loop

[2065] Step 1:

[2066] The server re-evaluates the user's utterance and generates specific feedback.

[2067] The server sends the generated feedback message to the terminal as text data.

[2068] Step 2:

[2069] The (device) displays the feedback message it received on the screen.

[2070] The user reviews the feedback message and understands the areas for improvement.

[2071] (Example 2)

[2072] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2073] Conventional customer service training systems could understand user intent and generate responses using speech recognition and natural language processing technologies, but they struggled to respond while considering user emotions. Furthermore, there were few means to evaluate the effectiveness of the training, making it difficult to efficiently improve user performance and skills. Additionally, training based on specific scenarios was limited, making it difficult to simulate real-world customer service situations.

[2074] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[2075] In this invention, the server includes means for converting user speech to text using a speech recognition engine, means for analyzing text data using natural language processing technology to understand user intent, means for generating a response based on intent and providing the generated response as text data, means for converting text data to speech and playing it back to the user, means for analyzing speech and text data to recognize user emotions, means for adjusting responses based on emotions to generate more appropriate dialogue, means for providing an interface for selecting persona information, means for preparing dialogue patterns based on the selected persona, means for starting a scenario based on the dialogue pattern and generating a speech response corresponding to the scenario, means for recording and evaluating user utterances, means for calculating points based on evaluation and updating points, and means for displaying points and performance. This enables the provision of appropriate responses that take emotions into account, improves the quality of dialogue, and allows for evaluation of user skills and confirmation of performance. Furthermore, by realizing training based on specific scenarios, it becomes possible to simulate situations that closely resemble actual customer service scenarios.

[2076] A "speech recognition engine" is a technology that takes speech data as input and converts it into text data.

[2077] "Natural language processing" is a technology that analyzes text data, understands human language, and generates appropriate responses.

[2078] A "server device" is a computer system that provides data processing and storage functions and operates in conjunction with client devices.

[2079] A "terminal device" is a device that a user directly operates and that provides interface functions while communicating with a server.

[2080] An "emotion engine" is a technology that analyzes voice and text data to estimate a user's emotional state.

[2081] A "persona" is a fictional character based on a specific scenario or user group, and is the subject of simulation in a training system.

[2082] A "dialogue pattern" is a definition of the expected sequence of questions and answers in a particular scenario.

[2083] "Evaluation criteria" are standards or standards used to evaluate user performance based on the content of user utterances and emotional state.

[2084] "Feedback" refers to evaluation comments and advice provided regarding a user's performance.

[2085] System Overview

[2086] This invention relates to a customer service training system that combines voice dialogue AI technology and emotion recognition. This system recognizes the user's voice and understands their intentions based on natural language processing technology, as well as recognizing the user's emotions using an emotion engine. Based on this information, it generates appropriate responses to improve skills.

[2087] Hardware and software configuration

[2088] This system consists of the following components:

[2089] Server equipment

[2090] Speech recognition engine: Converts speech data into text (e.g., Google Cloud Speech-to-Text)

[2091] Natural language processing engine: Analyzes text data and understands user intent (e.g., GPT-3).

[2092] Emotion engine: Analyzes user emotions based on voice and text data (e.g., Microsoft Azure Emotion API)

[2093] Response generation algorithm: Generates appropriate responses based on the user's intent and emotions.

[2094] Evaluation and point calculation system: Evaluates the user's utterances and calculates points.

[2095] Database: Stores user data, persona information, dialogue patterns, and sentiment data.

[2096] Terminal device

[2097] Microphone and speaker: Used for audio input and output.

[2098] Speech synthesis engine: Converts text data into speech (e.g., Amazon Polly)

[2099] Point and feedback display interface: Displays points and feedback to the user.

[2100] Persona Selection Interface: An interface for users to select a persona.

[2101] Specific implementations of the system

[2102] Users of this system use a terminal device to begin customer service training, including emotion recognition. Specific embodiments of the present invention are shown below.

[2103] Voice recording and recognition

[2104] The user presses the recording start button on the terminal device.

[2105] The device uses the microphone to record the user's voice.

[2106] The device sends the recorded audio data to the server.

[2107] The server uses a speech recognition engine to convert the audio data into text data.

[2108] Natural language processing and response generation

[2109] The server analyzes the text data using a natural language processing engine to understand the user's intent.

[2110] The server generates an appropriate response based on the user's intent and sends that response as text data to the terminal.

[2111] The device converts text data into speech using a speech synthesis engine and plays the response back to the user.

[2112] emotion recognition

[2113] The server passes voice and text data to the emotion engine, which then analyzes the user's emotions.

[2114] The server adjusts its response based on emotional data to generate more appropriate dialogue.

[2115] Persona-based training

[2116] The user selects a persona (e.g., "a customer with a complaint") on the terminal device.

[2117] The device sends the selected persona information to the server.

[2118] The server references persona-based dialogue patterns and prepares the scenario.

[2119] The server generates an initial question based on the scenario and sends it to the terminal as text data.

[2120] The device converts text data into speech and plays it back to the user.

[2121] Rating and point system

[2122] The device records the user's speech in real time and sends it to the server.

[2123] The server scores the spoken content and emotional data based on evaluation criteria.

[2124] The server calculates points based on the evaluation results and updates the user's performance database.

[2125] The server generates a feedback message and sends it to the terminal.

[2126] The device displays and provides a feedback message to the user.

[2127] Specific usage examples

[2128] Product description practice

[2129] The user selects a persona for the "product description" and starts the scenario.

[2130] The device starts recording and says, "Could you please explain this new product?"

[2131] The user says, "This product uses the latest technology..."

[2132] The server converts the audio to text, analyzes it, and then generates the question, "Could you please tell me the specific usage instructions?"

[2133] The server recognizes emotions from the user's voice and generates emotionally appropriate feedback.

[2134] The device plays back questions and feedback via voice, continuing the conversation.

[2135] Practice in handling complaints

[2136] The user selects the "complaint handling" persona and starts the scenario.

[2137] The device starts recording and says, "This product arrived damaged, so could I request a replacement?"

[2138] The user says, "I am very sorry, I will take care of it immediately."

[2139] The server converts the audio into text, analyzes it, and then generates the question, "Please tell me the specific steps you will take."

[2140] The server uses an emotion engine to recognize the user's anger level and generates additional responses to calm them down.

[2141] The device plays back the question and any additional responses via audio, continuing the conversation.

[2142] Examples of prompts for generative AI models

[2143] "Please provide a detailed explanation of the new product."

[2144] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2145] Step 1:

[2146] Audio recording and transfer

[2147] The user presses the recording start button on the terminal device. The input is the user's voice.

[2148] The device uses the microphone to record the user's voice. The output is the recorded audio data.

[2149] The device sends the recorded audio data to the server. The audio data is transferred from the device to the server.

[2150] Step 2:

[2151] Text conversion of audio data

[2152] The server uses a speech recognition engine to convert audio data into text data. The input is recorded audio data.

[2153] A speech recognition engine (such as Google Cloud Speech-to-Text) analyzes the audio data and converts it into text data. The output is text data.

[2154] Step 3:

[2155] Natural language processing and intent analysis

[2156] The server analyzes text data using a natural language processing engine. The input is text data.

[2157] A natural language processing engine (such as GPT-3) analyzes text data to understand the user's intent. The output is data that reflects the user's intent.

[2158] Step 4:

[2159] Response generation

[2160] The server generates an appropriate response based on the user's intent. The input is parsed intent data.

[2161] The response generation algorithm creates a text response tailored to the user's intent. The output is the text data of the response.

[2162] Step 5:

[2163] Speech and playback of responses

[2164] The server sends the generated text data to the terminal. The input is the response text data.

[2165] The device converts text data into speech using a speech synthesis engine (such as Amazon Polly). The output is the generated speech data.

[2166] The terminal plays the generated audio to the user. The audio data is provided to the user through the audio output device.

[2167] Step 6:

[2168] emotion recognition

[2169] The server passes audio and text data to the emotion engine. The input consists of audio and text data.

[2170] An emotion engine (such as the Microsoft Azure Emotion API) analyzes the data and recognizes the user's emotions. The output is emotion data.

[2171] The server adjusts its response based on sentiment data and generates a more appropriate dialogue. Adjusted response data is then generated.

[2172] Step 7:

[2173] Persona-based training begins.

[2174] The user selects a persona (e.g., "Customer with a complaint") on the terminal device. Persona information is then entered.

[2175] The device sends the selected persona information to the server. The input is the selected persona data.

[2176] The server references persona-based dialogue patterns and prepares a scenario. The output is scenario data.

[2177] The server generates the first question based on the scenario and sends it to the terminal as text data. The input is the scenario data.

[2178] The device converts text data into speech and plays it back to the user. The output is audio data.

[2179] Step 8:

[2180] Recording and evaluation of spoken content

[2181] The terminal records the user's speech in real time and sends it to the server. The input is the user's speech.

[2182] The server scores the utterance content and emotional data based on evaluation criteria. The input consists of the utterance content and emotional data.

[2183] The server calculates points based on the evaluation results and updates the user's performance database. The output is the updated performance data.

[2184] Step 9:

[2185] Generating and displaying feedback

[2186] The server generates a feedback message and sends it to the terminal. The input is the evaluation result.

[2187] The device displays and provides a feedback message to the user. The output is the feedback message.

[2188] (Application Example 2)

[2189] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2190] Conventional customer service training systems only utilize dialogue based on speech recognition and natural language processing technologies, and are unable to recognize user emotions and provide appropriate responses and feedback based on them. Therefore, it was difficult to effectively train staff in the ability to understand user emotions and respond appropriately in real-world customer service situations. Furthermore, the inability to conduct real-time training using cutting-edge technologies such as smart glasses meant there was a lack of means to improve training efficiency.

[2191] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting the user's voice into text using a speech recognition engine, means for analyzing the text data using natural language processing technology to understand the user's intent, and means for providing emotional recognition engines that analyze the user's emotions and provide emotionally appropriate responses and feedback messages. This makes it possible to recognize the user's voice in real time, generate responses based on their intent and emotions, and perform training through smart glasses.

[2192] A "speech recognition engine" is a technology used to convert speech data into text data.

[2193] "Natural language processing technology" is an information processing technology that analyzes text data to understand the user's intent.

[2194] An "emotion recognition engine" is a technology that analyzes a user's emotions based on voice data and text data.

[2195] A "server device" is a computer system that performs various functions such as data processing and analysis, response generation, and feedback provision.

[2196] A "terminal device" is a user device that communicates with a server device and performs functions such as audio recording and playback, and interface display.

[2197] "Text data" refers to character information converted by a speech recognition engine.

[2198] A "response generation algorithm" is a program that creates an appropriate response based on the user's intentions and emotions.

[2199] "Smart glasses" are devices that, when worn by the user, provide visual information and interactive interfaces.

[2200] A "feedback message" is a message generated based on the user's training results and emotions, intended for evaluation and improvement.

[2201] A "persona" is a virtual character or role used to simulate a specific dialogue scenario or situation.

[2202] This invention is a system that combines voice dialogue AI technology and emotion recognition technology to provide real-time customer service training through smart glasses. The following describes embodiments for carrying out this invention.

[2203] System Configuration

[2204] 1. Server equipment

[2205] Speech recognition engine

[2206] Natural Language Processing Technology

[2207] Emotion recognition engine

[2208] Response generation algorithm

[2209] Evaluation and point calculation system

[2210] Database (user data, persona information, dialogue patterns, sentiment data)

[2211] 2. Terminal device

[2212] Smart Glasses

[2213] Microphone and speaker

[2214] Speech synthesis engine

[2215] Point and feedback display interface

[2216] Persona Selection Interface

[2217] Program details

[2218] The server device performs the following steps when a user uses a terminal device (smart glasses) to conduct customer service training.

[2219] 1. Voice recording and recognition

[2220] The server device receives the user's voice data collected by the terminal device and converts it into text data using a speech recognition engine.

[2221] 2. Natural Language Processing and Response Generation

[2222] The server device analyzes the converted text data using natural language processing technology to understand the user's intent. Next, it generates an appropriate response based on that intent using a response generation algorithm and sends it to the terminal device as text data.

[2223] 3. Emotion recognition and adjustment of responses

[2224] The server device passes voice and text data to the emotion recognition engine, which analyzes the user's emotions. Based on the analyzed emotion data, it adjusts its response and generates a more appropriate feedback message.

[2225] 4. Real-time training via smart glasses

[2226] The smart glasses, which are the terminal device, convert the received response into speech using a speech synthesis engine and play it back to the user. They also visually display the evaluation results to the user through a points and feedback display interface.

[2227] Devices and Software

[2228] The server device has a program installed to analyze the collected data and generate appropriate responses. The hardware and software used are as follows:

[2229] Speech recognition engine: Google Speech Recognition API

[2230] Natural Language Processing Techniques: Hugging Face's Transformers Library

[2231] Emotion Recognition Engine: Emotion Analysis Pipeline using the Transformers Library

[2232] Speech synthesis engine: Google Speech API

[2233] Smart glasses: Devices that provide visual and auditory information when worn by the user.

[2234] Specific example

[2235] When a store clerk speaks through smart glasses and says, "Could you tell me more about this product?", the system converts the speech into text and analyzes the clerk's emotions. If the clerk is excited, the system responds, "That's a great suggestion! Do you have any further questions?", and if the clerk is confused, it responds, "I'm sorry. I'll do my best to resolve the issue." The system always provides guidance that is appropriate to the clerk's emotions.

[2236] Example of a prompt:

[2237] User's utterance:

[2238] "Please tell me more about this product."

[2239] Example system response:

[2240] "That's a great suggestion! Do you have any further questions?"

[2241] User's emotional state: positive

[2242] Expected system response:

[2243] "That's a great suggestion! Do you have any further questions?"

[2244] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2245] Step 1:

[2246] The user puts on the smart glasses and starts training mode. The user selects the persona they want to train (e.g., "product description" or "customer complaint handling") through the persona selection interface. The device sends the selected persona information to the server.

[2247] Step 2:

[2248] The server retrieves appropriate dialogue patterns from the database based on the selected persona and prepares a training scenario. The server generates an initial question or situation description and sends the text data to the terminal.

[2249] Step 3:

[2250] The device converts the received text data into speech using a speech synthesis engine and plays it back to the user. The user then inputs the next action or response via voice.

[2251] Step 4:

[2252] The terminal records the user's voice input and sends it to the server as audio data. The server uses a speech recognition engine to convert the audio data into text data. The voice input is a specific question, such as "Please tell me more about this product."

[2253] Step 5:

[2254] The server analyzes the converted text data using natural language processing techniques to understand the user's intent, such as a specific request like "Please tell me the details of the product." Based on this analysis, the server generates the next appropriate response using a response generation algorithm.

[2255] Step 6:

[2256] The server analyzes the user's voice and text data using an emotion recognition engine to identify the user's emotional state. For example, it can determine whether the user is excited or confused.

[2257] Step 7:

[2258] The server adjusts its response based on the emotions identified by the emotion recognition engine, generating a more appropriate feedback message. For example, if the user is confused, it might respond with, "We're sorry. We'll do our best to resolve the issue." This response is then sent to the terminal as text data.

[2259] Step 8:

[2260] The device converts the received text data back into speech using a speech synthesis engine and plays it back to the user. This allows the user to receive feedback in real time.

[2261] Step 9:

[2262] The server passes the entire conversation record to the evaluation system, which scores the user's performance. Based on the evaluation results, points are calculated and the user's performance database is updated.

[2263] Step 10:

[2264] The device displays points and evaluation results to the user through a feedback display interface. For example, it provides evaluations using specific numerical values, such as "You earned 10 points in today's training."

[2265] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2266] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2267] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[2268] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2269] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[2270] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[2271] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[2272] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[2273] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[2274] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[2275] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[2276] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[2277] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[2278] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2279] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[2280] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[2281] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[2282] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[2283] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[2284] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[2285] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[2286] The following is further disclosed regarding the embodiments described above.

[2287] (Claim 1)

[2288] A means of converting user speech to text using a speech recognition engine,

[2289] A means of analyzing text data using natural language processing technology to understand user intent,

[2290] A server device that generates a response based on intent and provides the generated response as text data,

[2291] A terminal device that converts text data into speech and plays it back to the user,

[2292] A system that includes this.

[2293] (Claim 2)

[2294] A terminal device that provides an interface for selecting persona information,

[2295] A server device that prepares dialogue patterns based on the selected persona,

[2296] A server device that initiates a scenario based on a dialogue pattern and generates a voice response according to the scenario,

[2297] The system according to claim 1, including the following:

[2298] (Claim 3)

[2299] A means of recording and evaluating user speech content,

[2300] A server device that calculates points based on the evaluation and updates the points,

[2301] A terminal device that displays points and scores,

[2302] The system according to claim 1, including the following:

[2303] "Example 1"

[2304] (Claim 1)

[2305] A speech recognition means that converts user speech into text data,

[2306] A natural language processing method that analyzes text data and understands the user's intent,

[2307] A response generation means that generates a response based on the user's intent and provides the generated response as text data,

[2308] A speech synthesis method that converts text data into speech and plays it back to the user,

[2309] A system that includes this.

[2310] (Claim 2)

[2311] A means of providing an interface for selecting persona information,

[2312] A means of preparing dialogue patterns based on the selected persona,

[2313] A means for initiating a scenario based on a dialogue pattern and generating a voice response corresponding to the scenario,

[2314] The system according to claim 1, including the following:

[2315] (Claim 3)

[2316] A means of recording and evaluating user speech content,

[2317] A means of calculating points based on evaluation and updating points,

[2318] A means of displaying points and scores,

[2319] The system according to claim 1, including the following:

[2320] "Application Example 1"

[2321] (Claim 1)

[2322] A means of converting user speech to text using a speech recognition engine,

[2323] A means of analyzing text data using natural language processing technology to understand user intent,

[2324] A server device that generates a response based on intent and provides the generated response as text data,

[2325] A terminal device that converts text data into speech and plays it back to the user,

[2326] A means of capturing user actions with a camera and evaluating them in sync with speech content,

[2327] A means for generating a response based on the user's utterance using a generative AI model,

[2328] A means of providing the generated response to the user as evaluation and feedback,

[2329] A means of storing the content of the utterance and the evaluation results in a database,

[2330] A system that includes this.

[2331] (Claim 2)

[2332] A terminal device that provides an interface for selecting persona information,

[2333] A server device that prepares dialogue patterns based on the selected persona,

[2334] A server device that initiates a scenario based on a dialogue pattern and generates a voice response according to the scenario,

[2335] A means of capturing and recording the user's actions and speech in real time using a camera and microphone built into smart glasses,

[2336] The system according to claim 1, including the following:

[2337] (Claim 3)

[2338] A means of recording and evaluating user speech content,

[2339] A server device that calculates points based on the evaluation and updates the points,

[2340] A means of providing feedback to users using a terminal device that displays points and scores,

[2341] A means for automatically generating prompt sentences for a generative AI model and optimizing dialogue patterns based on the user's dialogue history,

[2342] The system according to claim 1, including the following:

[2343] "Example 2 of combining an emotion engine"

[2344] (Claim 1)

[2345] A means of converting user speech to text using a speech recognition engine,

[2346] A means of analyzing text data using natural language processing technology to understand user intent,

[2347] A server device that generates a response based on intent and provides the generated response as text data,

[2348] A terminal device that converts text data into speech and plays it back to the user,

[2349] A means of recognizing user emotions by analyzing voice and text data,

[2350] A means of adjusting responses based on emotions and generating more appropriate dialogue,

[2351] A terminal device that provides an interface for selecting persona information,

[2352] A server device that prepares dialogue patterns based on the selected persona,

[2353] A server device that initiates a scenario based on a dialogue pattern and generates a voice response according to the scenario,

[2354] A means of recording and evaluating user speech content,

[2355] A server device that calculates points based on the evaluation and updates the points,

[2356] A terminal device that displays points and scores,

[2357] A system that includes this.

[2358] (Claim 2)

[2359] A means of converting speech into text data if speech recognition is successful,

[2360] A means for analyzing text data using natural language processing and generating a response,

[2361] A means for analyzing a user's emotions using an emotion engine and generating corresponding dialogue responses,

[2362] The system according to claim 1, including the following:

[2363] (Claim 3)

[2364] A server device that provides a persona selection interface and references dialogue patterns based on persona inform...

Claims

1. A means of converting user speech to text using a speech recognition engine, A means of analyzing text data using natural language processing technology to understand user intent, A server device that generates a response based on intent and provides the generated response as text data, A terminal device that converts text data into speech and plays it back to the user, A system that includes this.

2. A terminal device that provides an interface for selecting persona information, A server device that prepares dialogue patterns based on the selected persona, A server device that initiates a scenario based on a dialogue pattern and generates a voice response according to the scenario, The system according to claim 1, including the following:

3. A means of recording and evaluating user speech content, A server device that calculates points based on the evaluation and updates the points, A terminal device that displays points and scores, The system according to claim 1, including the following:

Images