System

Abstract

A system is provided.SOLUTION: A means for a user to access a dialogue simulator system and select a scenario, a means for a server to read data corresponding to the selected scenario and customize the scenario, a means for the server to generate an AI character and transmit setting data of the character to a terminal, and a means for the terminal to receive utterance data for the user to start a dialogue; A system for providing an interaction simulator including an expression and an intonation by means for transmitting to a server, means for the server to analyze speech data of a user, generate an appropriate response, and transmit the response to a terminal, means for the server to analyze speech contents, an expression, and an intonation of a way of speaking of the user and generate a feedback, means for the server to transmit the generated feedback to the terminal and provide the feedback to the user, and means for the server to store an interaction history and the feedback of the user and utilize the interaction history and the feedback for future learning.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional vocational training requires the allocation of personnel as role-playing partners, and depending on their level of proficiency, realistic interactions can be difficult. Furthermore, new employees often have few opportunities to learn meeting behavior and presentation skills, resulting in insufficient practice. Furthermore, it is difficult to provide feedback on elements that are difficult to verbalize, such as facial expressions and intonation, making it difficult to effectively improve skills. To solve these issues, a system is needed that allows for efficient and effective role-playing and presentation practice. [Means for solving the problem]

[0005] The dialogue simulator system of the present invention provides a means for a user to access the dialogue simulator and select a scenario, and the server loads data corresponding to the selected scenario to customize the scenario. Furthermore, the server generates an AI character and transmits its configuration data to the terminal. When the user begins a dialogue, the terminal transmits utterance data to the server, which analyzes the data, generates an appropriate response, and transmits it to the terminal. The server also analyzes the user's utterance content, facial expressions, and intonation to generate feedback, which is then transmitted to the terminal and provided to the user. Additionally, the server has a means for saving the user's dialogue history and feedback and utilizing it for future learning. These means enable effective skill improvement through a realistic dialogue experience, including facial expressions and intonation.

[0006] The "Dialogue Simulator System" is a system that allows users to role-play based on a set scenario and improve their skills through dialogue with AI characters.

[0007] A "user" is a person who uses the dialogue simulator system to role-play and practice skills.

[0008] The "server" is a computer system that performs the central processing of the dialogue simulator system, reads the data of the selected scenario, generates AI characters, and provides analysis and feedback.

[0009] A "terminal" is a device (e.g., PC, tablet, smartphone) through which a user accesses the dialogue simulator system and receives dialogue and feedback.

[0010] A "scenario" refers to the content of exercises selected by the user in the dialogue simulator system, and includes specific settings such as sales role-playing, meeting practice, and presentation skill improvement.

[0011] An "AI character" is a virtual character generated within the dialogue simulator system to converse with the user, and its appearance, personality, intonation, and facial expression patterns are set.

[0012] "Utterance data" refers to voice or text information uttered by a user when engaging in a dialogue.

[0013] A "response" refers to an appropriate response that the server generates by analyzing the user's speech data.

[0014] "Feedback" is information that includes suggestions for improvement and advice that is generated by the server based on the results of analyzing the user's speech content, facial expressions, and intonation, and is provided to the user.

[0015] The "dialogue history" is a record of a series of dialogues that a user has conducted using the dialogue simulator system. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The dialogue simulator system of the present invention begins when a user accesses the dialogue simulator system and selects a scenario. Once the user selects a scenario, the terminal transmits the selected scenario information to the server. The server reads data corresponding to the selected scenario from the database and customizes the scenario based on the user's past practice history.

[0038] The server then generates an AI character, which is configured to interact with the user, and the configuration data includes the character's appearance, personality, intonation, and facial expression patterns. The generated configuration data is sent to the device, which then displays it to the user.

[0039] When a user initiates a conversation, the device records the user's speech in real time and sends it to the server. The server analyzes the speech data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends this response data to the device. The device then provides a response to the user through an AI character, whose facial expressions and intonation also change according to the situation.

[0040] The server then analyzes the user's speech, facial expressions, and intonation in real time and generates feedback based on these. The feedback is then sent to the device and provided to the user in voice or text format. This feedback includes specific advice and suggestions for improvement.

[0041] Finally, the server stores the user's interaction history and feedback in a database, which is used for future practice and to analyze the user's progress and trends to customize the next scenario.

[0042] Specific examples

[0043] For example, if a user selects a scenario for improving presentation skills, the device sends "Improve presentation skills" to the server, and the server loads and customizes the scenario based on the associated template and past practice data.

[0044] To start a presentation, the user says, "Today I'm going to present about our new product." The device records this speech and sends it to the server. The server analyzes this information, generates a response such as, "Tell me more about your new product," and sends it back to the user via the device.

[0045] While the user continues the presentation, the server analyzes the user's speech content and intonation and generates appropriate feedback in real time. For example, after the presentation, specific feedback such as "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more" is provided.

[0046] Through these steps, users can effectively improve their presentation skills. This system provides an environment that is as close as possible to real conversations, and supports users in improving their skills.

[0047] The processing flow will be explained below.

[0048] Step 1:

[0049] The user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface.

[0050] Step 2:

[0051] The user selects the scenario of their choice (sales role-play, meeting practice, presentation skills improvement, etc.).

[0052] Step 3:

[0053] The terminal transmits the user's scenario selection information to the server.

[0054] Step 4:

[0055] The server reads data corresponding to the selected scenario from the database.

[0056] Step 5:

[0057] The server checks the user's past practice history and customizes the scenario.

[0058] Step 6:

[0059] The server generates configuration data for the AI ​​character, including the character's appearance, personality, intonation, and facial expression patterns.

[0060] Step 7:

[0061] The server sends the AI ​​character configuration data generated by the server to the device.

[0062] Step 8:

[0063] The device displays the AI ​​character to the user, preparing the user to begin interaction.

[0064] Step 9:

[0065] The user initiates the interaction and speaks the first word (e.g., beginning a presentation).

[0066] Step 10:

[0067] The device records the user's voice input and transmits the data to the server in real time.

[0068] Step 11:

[0069] The server analyzes the voice data and uses natural language processing to understand the user's intent.

[0070] Step 12:

[0071] The server generates an appropriate response based on the user's intentions, taking into account the character's personality and the scenario content.

[0072] Step 13:

[0073] The server generates a response and sends it to the terminal.

[0074] Step 14:

[0075] The device responds to the user in voice or text, changing the AI ​​character's facial expression and intonation.

[0076] Step 15:

[0077] The user then makes the next utterance and the dialogue continues. The device again records the voice input and sends it to the server.

[0078] Step 16:

[0079] The server analyzes the user's speech in real time and generates feedback based on the speech content, facial expressions, and speaking intonation.

[0080] Step 17:

[0081] The server generates feedback and sends it to the device.

[0082] Step 18:

[0083] The terminal provides the generated feedback to the user in real time by voice or text.

[0084] Step 19:

[0085] The server stores the user's interaction history and feedback in a database.

[0086] Step 20:

[0087] The server analyzes the user's learning status based on the stored data and generates data to customize the content of the next practice session.

[0088] Step 21:

[0089] The device will then display a report of the results to the user, including a score, specific areas for improvement, and suggestions for the next practice session.

[0090] Through the above steps, users can effectively improve their skills through interaction.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] Current dialogue simulation systems suffer from monotonous reactions to user utterances and a lack of appropriate feedback that takes into account the user's past practice history. This makes it difficult for users to achieve sufficient learning results, particularly in finer details such as pronunciation, intonation, and facial expressions, which can slow down improvement. Furthermore, the lack of advanced conversational simulation using generative AI models limits the user experience.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for a terminal to transmit the selected scenario information to the server; means for the server to read scenario data and the user's past practice history from a database and customize the scenario; means for the server to generate an AI character and transmit character setting data to the terminal; means for the terminal to record utterance data for the user to start a dialogue in real time and transmit it to the server; means for the server to analyze the voice data, understand the user's intention using natural language processing and generate an appropriate response; means for the server to transmit the generated response data to the terminal and provide the response to the user through the AI ​​character; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation, generate feedback, and transmit the generated feedback to the terminal and provide it to the user; and means for the server to store the dialogue history and feedback in a database and use it for future learning. This allows users to practice in an environment that more closely resembles actual dialogue and receive detailed feedback to improve their skills.

[0096] A "user" is a subject who uses the dialogue simulator system to select a scenario and engage in dialogue.

[0097] The "dialogue simulator system" is a system that simulates dialogue based on a scenario selected by the user and provides reactions and feedback from an AI character.

[0098] "Scenario" refers to a practice or scene setting selected by the user in the dialogue simulator system.

[0099] A "terminal" is a hardware device (for example, a personal computer or smartphone) that allows a user to access and operate the dialogue simulator system.

[0100] The "server" is the central component of the dialogue simulator system, and is a computer that loads and customizes scenario data, generates AI characters, analyzes user speech data, and generates responses.

[0101] "Scenario data" refers to data such as templates, storylines, and example questions and answers related to each scenario in the dialogue simulator system.

[0102] The "database" is a data storage system for storing scenario data, the user's past practice history, dialogue history, feedback, and so on.

[0103] "Customization" is the process of adaptively modifying and adjusting a scenario, taking into account the user's past practice history and other relevant data.

[0104] An "AI character" is an artificial intelligence-based character (avatar) generated in a dialogue simulator system to engage in dialogue with a user.

[0105] "Character setting data" refers to data that includes information such as the AI ​​character's appearance, personality, intonation, and facial expression patterns.

[0106] "Utterance data" refers to recorded data of voice utterances made by a user during a conversation.

[0107] "Natural language processing" is a technology that allows an AI character to understand what the user is saying and generate an appropriate response based on that.

[0108] "Response data" is data that includes a response or reaction generated by the server in response to the content of the user's utterance.

[0109] "Feedback" is evaluation information that includes advice and suggestions for improving the dialogue, and is generated based on the user's speech content, facial expressions, intonation, etc.

[0110] "Dialogue history" refers to a record of the dialogues that a user has had with the dialogue simulator system.

[0111] The dialogue simulator system of the present invention begins when a user accesses the dialogue simulator system and selects a scenario. The user accesses the system from a browser on a PC or smartphone, logs in, and then selects a scenario that suits their purpose from a list of scenarios provided. For example, the user can select the scenario "Improve Presentation Skills."

[0112] Once a selection is made, the device sends this information to the server via an HTTP request (for example, the POST method). At this time, the data sent includes the user ID and the selected scenario ID. On the server side, the corresponding scenario data is read from a database (for example, MySQL or PostgreSQL) based on the received scenario ID. At the same time, the user's past practice history is obtained and the scenario is customized based on this. This customization reflects the themes the user has previously experienced and the points that should be emphasized.

[0113] Next, the server generates an AI character. This AI character is generated in real time using, for example, a custom Unity API. The generated character's appearance, personality, intonation, facial expression patterns, etc. are configured. This configuration data is sent to the device in JSON format, and the device uses WebGL or WebRTC technology to display it on the screen.

[0114] When a conversation begins, the device records the user's speech in real time and performs noise cancellation and echo removal to obtain high-quality audio data. This audio data is sent to a server, where it is converted into text using speech recognition technology such as the Google Cloud Speech-to-Text API. The server then analyzes the user's intent using natural language processing with a generative AI model (e.g., BERT or GPT-3). Based on the results of this analysis, the server generates an appropriate response and sends the response data back to the device.

[0115] Based on the received response data, the device uses voice synthesis technology such as the Web Speech API to have the AI ​​character respond, changing facial expressions and intonation along with the response to achieve a more natural dialogue.

[0116] Furthermore, the server analyzes the user's speech content, facial expressions, and speaking intonation in real time, and provides specific feedback based on the results. This feedback may include detailed advice such as, "The product explanation section was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more." This feedback is sent from the server to the device, which then provides it to the user. This process is carried out using emotion analysis tools such as Microsoft Azure's Emotion API.

[0117] Finally, the server stores all interaction history and feedback in a database, which can be used for future practice sessions, to analyze the user's progress, and to provide more customized scenarios.

[0118] Examples of specific examples and prompts

[0119] For example, if a user selects the "Improve Presentation Skills" scenario, the scenario will include the following prompt:

[0120] "Please tell us three features of your new product."

[0121] "Please suggest appropriate responses to the audience questions."

[0122] "Please tell me how to effectively summarize at the end of a presentation."

[0123] When a user says, "Today I'm going to give a presentation about our new product," the device's microphone records this speech and sends it to the server, which then responds with, "Please tell us more about your new product." After the presentation, the server analyzes the user's speech and provides feedback such as, "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more."

[0124] This system allows users to improve even the finer details of pronunciation, intonation, and facial expressions through dialogue.

[0125] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0126] Step 1:

[0127] This is the procedure in which a user accesses the dialogue simulator system and selects a scenario.

[0128] Input: User login information, system access, scenario selection request.

[0129] Data processing and calculation: User authentication, scenario list display, confirmation of selected scenario.

[0130] Output: Selected scenario ID, user ID.

[0131] Specific operation: A user logs in from a PC or smartphone browser, browses the list of scenarios, and then selects a scenario such as "Improve Presentation Skills."

[0132] Step 2:

[0133] This is a procedure in which the terminal transmits scenario selection information to the server.

[0134] Input: Selected scenario ID, user ID.

[0135] Data processing and calculation: Generating and sending HTTP requests (POST method, etc.).

[0136] Output: The scenario selection request sent to the server.

[0137] Specific operation: The device generates an HTTP request including the selected scenario ID and user ID and sends it to the server.

[0138] Step 3:

[0139] This is the procedure in which the server reads scenario data and the user's past practice history from the database and customizes the scenario.

[0140] Input: Scenario ID, User ID.

[0141] Data processing and calculation: Performing database queries, filtering and extracting past exercise history, and adaptive scenario customization.

[0142] Output: Customized scenario data.

[0143] Specific operation: The server reads the relevant scenario data and the user's past practice history from the MySQL database, and customizes the scenario based on the previous practice content.

[0144] Step 4:

[0145] This is the procedure in which the server generates an AI character and sends the character's setting data to the terminal.

[0146] Input: Customized scenario data.

[0147] Data processing and calculation: Calling the AI ​​character generation module and generating setting data.

[0148] Output: AI character configuration data (JSON format).

[0149] How it works: The server uses Unity's custom API to generate a character, create settings such as appearance, personality, intonation, and facial expression patterns, and send them to the device.

[0150] Step 5:

[0151] This is the procedure in which the device displays an AI character and begins a dialogue with the user.

[0152] Input: AI character configuration data.

[0153] Data processing and calculation: Reading setting data, using the screen drawing library.

[0154] Output: An AI character displayed on the screen.

[0155] Specific operation: The device displays an AI character on the screen based on the configuration data received using WebGL and WebRTC technology, and the conversation begins when the user presses the "Start" button.

[0156] Step 6:

[0157] This is a procedure in which the terminal records the user's speech data and transmits it to the server.

[0158] Input: User utterance.

[0159] Data processing and calculation: voice recording, noise cancellation, echo removal.

[0160] Output: Preprocessed audio data.

[0161] Specific operation: When a user speaks to an AI character, the device's microphone records the voice, performs noise cancellation and echo removal in real time, and then sends the voice data to the server.

[0162] Step 7:

[0163] This is the procedure where the server analyzes the voice data and generates an appropriate response.

[0164] Input: Preprocessed audio data.

[0165] Data processing and calculation: Speech-to-Text, natural language processing (BERT and GPT-3).

[0166] Output: Response data (text format).

[0167] How it works: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text, then uses a natural language processing model to analyze the user's intent and generate an appropriate response.

[0168] Step 8:

[0169] This is the procedure by which the device displays a response to the user through an AI character.

[0170] Input: Response data from the server (text format).

[0171] Data processing and calculation: voice synthesis, facial animation generation.

[0172] Output: Voice responses and facial expressions by the AI ​​character.

[0173] How it works: The device uses the Web Speech API to convert the response into voice and responds to the user along with the AI ​​character's facial expressions, resulting in a more natural dialogue.

[0174] Step 9:

[0175] This is a procedure in which the server analyzes the user's speech and generates real-time feedback.

[0176] Input: User utterance and parsed results.

[0177] Data processing and calculation: Sentiment analysis, speech evaluation, and extraction of areas for improvement.

[0178] Output: Feedback data.

[0179] Specific operation: The server uses Microsoft Azure's Emotion API and emotion analysis tools to evaluate the user's speech and generate feedback including specific improvements and advice.

[0180] Step 10:

[0181] This is a procedure in which the terminal provides feedback to the user.

[0182] Input: Feedback data.

[0183] Data processing and calculation: Display of feedback contents.

[0184] Output: The displayed feedback.

[0185] Specific operation: The device provides the received feedback data to the user by voice or text, using a text box displayed on the screen or a speaker for voice feedback.

[0186] Step 11:

[0187] This is the procedure by which the server stores the dialogue history and feedback in a database.

[0188] Input: Dialogue history, feedback data.

[0189] Data processing and calculation: Writing to database, saving data.

[0190] Output: Stored interaction history and feedback.

[0191] What it does: The server stores all interaction history and feedback in a MySQL database, which is used to customize the scenario for the next session.

[0192] (Application example 1)

[0193] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0194] Previously, training to improve customer service skills in brick-and-mortar stores required effort and cost to deal with real customers, and had the problem of not being able to receive immediate feedback. Furthermore, there was a lack of opportunities for new staff to practice interacting with real customers, making it difficult for them to improve their skills. For this reason, there is a need for a system that can improve customer service skills while receiving effective, real-time feedback.

[0195] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0196] In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and send character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and send it to the server; means for the server to analyze the user's utterance data to generate an appropriate response and send it to the terminal; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation and generate feedback; means for the server to send the generated feedback to the terminal and provide it to the user; means for the server to save the user's dialogue history and feedback and use it for future learning; means for the terminal to function as a customer service simulator and enable brick-and-mortar store staff to improve their customer service skills; and means for providing interactive feedback to the user. This enables brick-and-mortar store staff to improve their customer service skills by receiving feedback effectively and in real time.

[0197] A "dialogue simulator system" is a system that allows users to improve specific skills or knowledge by engaging in dialogue in a simulated format.

[0198] A "scenario" is a set of stories or conditions designed to simulate a particular situation or dialogue.

[0199] A "server" is a computer system that serves as the core of the dialogue simulator system accessed by users, and is a device that processes data and generates AI characters.

[0200] "Means for reading data and customizing a scenario" refers to a mechanism in which the server reads related information from a database for a scenario selected by a user and modifies it to suit the user's needs.

[0201] An "AI character" is a virtual character with artificial intelligence that is generated in a dialogue simulator to have an interactive conversation with a user.

[0202] "Configuration data" is data that defines the specific attributes and behavior of an AI character, such as their appearance, personality, intonation, and facial expression patterns.

[0203] "Utterance data" is data of the voice uttered by the user during a conversation.

[0204] A "reply" is a message that the server generates in response to an analysis of the user's speech data.

[0205] "Expressions" refer to the facial movements and emotional expressions shown by AI characters in the dialogue simulator.

[0206] "Intonation" refers to the rise and fall of the voice when speaking.

[0207] "Feedback" refers to information such as evaluations and areas for improvement provided based on the user's interactions and actions.

[0208] "Dialogue history" is a record of past dialogues between the user and the AI ​​character.

[0209] The "customer service simulator" is a system that allows store staff to simulate interactions with customers and improve their response skills.

[0210] "Interactive feedback" refers to dynamic evaluations and advice provided in real time in response to a user's actions.

[0211] This invention relates to a customer interaction simulator system that allows store staff to improve their customer interaction skills. The system starts when a user accesses the interaction simulator system and selects a scenario. Based on the selected scenario, the server loads data and customizes the scenario. The server then generates an AI character and sends its setting data to the terminal.

[0212] After the user initiates a dialogue, the device receives the user's speech data and sends it to the server. The server analyzes the user's speech data, generates an appropriate response using a generative AI model, and sends this response data to the device. The device then provides a response to the user through an AI character. At this time, the AI ​​character's facial expression and intonation also change according to the scenario. The server then analyzes the content of the user's speech, facial expression, and speaking intonation to generate feedback. The generated feedback is sent to the device and provided to the user.

[0213] Hardware and software used

[0214] Hardware:

[0215] Smart glasses (e.g., smart devices)

[0216] Head-mounted displays (e.g., mixed reality devices)

[0217] software:

[0218] Python (programming language)

[0219] speech_recognition (speech recognition library)

[0220] transformers (natural language processing library)

[0221] The server captures the user's voice input data and converts it into text using a speech recognition library. The converted data is then used by an AI character to generate an appropriate response using a natural language processing library. This response is then provided to the user via the device. The user's conversation is analyzed in real time and feedback is generated. This feedback includes specific advice and suggestions for improvement and is provided to the user interactively via the device.

[0222] Specific examples

[0223] For example, suppose a staff member at a brick-and-mortar store selects the "Basic Customer Response Scenario" and starts training. The staff member says, "Hello, is there anything you'd like to buy in our store?" The device records this speech, and the server analyzes the audio data. Based on the analysis results, the server generates a response such as "Hello, is there anything in particular you'd like to buy today?" and displays the response through the device.

[0224] The system then analyzes the speech and intonation of the staff member and provides feedback such as, "If you ask your question with a more emphasized intonation, it will be easier for the customer to understand." This feedback is provided interactively in real time, helping the staff member improve their skills.

[0225] In this way, store staff can effectively improve their customer service skills.

[0226] Example prompt sentence:

[0227] "Hello, are you looking for something in the store?"

[0228] "Is there anything in particular you're looking for today?"

[0229] "If you ask questions with a bit more emphasis on intonation, you'll be able to communicate more clearly to the customer."

[0230] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0231] Step 1:

[0232] A user accesses the dialogue simulator system and selects a scenario. As input, the user accesses the scenario selection interface from the device they use (smart glasses or a head-mounted display) and selects a specific scenario (e.g., "Basic Customer Interaction Scenario"). As output, the selected scenario information is sent to the server.

[0233] Step 2:

[0234] The server reads the data corresponding to the selected scenario and customizes the scenario. The input is the scenario selection information sent by the user, and based on this the server retrieves related information from the database. The data is processed by customizing the scenario based on the user's past practice history, and customized scenario data is generated as output.

[0235] Step 3:

[0236] The server generates an AI character and sends the character's configuration data to the device. The input is customized scenario data, and the AI ​​character's configuration (appearance, personality, intonation, facial expression patterns, etc.) is generated. The data is processed using a generative AI model to generate the character, and the configuration data is generated as output and sent to the device.

[0237] Step 4:

[0238] The user starts a conversation. The device receives the user's speech data and sends it to the server. The input is the user's voice data at the start of the conversation, which the device captures in real time. The data is then processed using a speech recognition library to convert the voice data into text data, and the converted text data is generated as output and sent to the server.

[0239] Step 5:

[0240] The server analyzes the user's speech data, generates an appropriate response, and sends it to the device. The converted text data is used as input, and a response is generated using a generative AI model based on this. Natural language processing is used to process the data, analyzing the user's intent, and an appropriate response data is generated as output and sent to the device.

[0241] Step 6:

[0242] The device provides the user with a response from an AI character. The response data sent from the server is input, and the device receives this. The data is processed by changing the AI ​​character's facial expression and intonation based on the settings, and the response is provided to the user visually and audibly as output.

[0243] Step 7:

[0244] The server analyzes the user's speech content, facial expressions, and speaking intonation to generate feedback. The input is the user's latest speech data and dialogue history, and data calculations are performed using an AI model based on this data. The output is feedback that includes specific advice and suggestions for improvement.

[0245] Step 8:

[0246] The server sends the generated feedback to the terminal and provides it to the user. The generated feedback data is input and sent to the terminal. The data is processed by adjusting the display format of the feedback, and the feedback is provided to the user in an interactive form as output.

[0247] Step 9:

[0248] The server stores the user's interaction history and feedback and uses it for future learning. The input is the latest interaction history and feedback data, and the data is processed and stored appropriately in a database. The output is used to customize future scenarios and improve the learning algorithm.

[0249] Example prompt sentence:

[0250] "Hello, are you looking for something in the store?"

[0251] "Is there anything in particular you're looking for today?"

[0252] "If you ask questions with a bit more emphasis on intonation, you'll be able to communicate more clearly to the customer."

[0253] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0254] The dialogue simulator system of the present invention is structured so that a user accesses the system and selects a scenario, and the entire system is processed mainly by the server. The specific flow of processing will be explained below.

[0255] When starting to use the system, the user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface. The user selects the desired scenario (e.g., sales role-play, meeting practice, presentation skill improvement, etc.), and the selection information is sent from the terminal to the server.

[0256] The server reads data corresponding to the selected scenario from the database and customizes the scenario based on the user's past practice history. Based on this customized scenario, the server generates configuration data for the AI ​​character and sends the configuration data to the device. The device displays the generated AI character to the user, ready for the user to begin interacting with it.

[0257] When a user initiates a dialogue and speaks, the device records the user's voice input and sends it to the server in real time. The server analyzes the voice data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends the response data to the device. The device responds to the user by changing the AI ​​character's facial expression and intonation.

[0258] Furthermore, the system uses an emotion engine to analyze the user's speech and facial expression data. The server uses the emotion engine to recognize the user's emotions and adjusts the response and feedback accordingly. For example, if the user is nervous, the AI ​​character will respond in an encouraging manner to calm the user.

[0259] The server also uses the emotion engine to generate feedback. It analyzes the user's speech content, facial expressions, and speaking intonation in real time, and generates specific feedback based on the analysis results. This feedback is sent to the device and provided to the user in the form of voice or text. The feedback includes specific advice and areas for improvement.

[0260] For example, suppose a user selects a scenario for improving presentation skills and begins a presentation. When the user says, "Today I'm going to give a presentation about our new product," the device records this speech and sends it to the server. The server analyzes this speech data and generates a response such as, "Tell me more about your new product." It then analyzes the user's emotions using an emotion engine and adjusts the tone and content of the response as needed.

[0261] After the presentation, the server analyzes the user's speech and intonation, and provides specific feedback such as, "The product explanation was particularly easy to understand in the presentation. However, it would be even better if you emphasized the intonation a little more." The feedback is also adjusted appropriately, taking into account the user's emotions.

[0262] Finally, the server stores the user's dialogue history and feedback in a database and uses it as data to personalize the next scenario. In this way, the dialogue simulator system of the present invention helps users improve their skills and provides a more effective training environment.

[0263] To implement the invention, a program is required to smoothly carry out a series of operations, including user selection, data analysis, emotion recognition, and feedback generation, allowing users to effectively improve their practical skills through realistic interactions.

[0264] The processing flow will be explained below.

[0265] Step 1:

[0266] The user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface.

[0267] Step 2:

[0268] The user selects the scenario of their choice (e.g., sales role-play, meeting practice, presentation skill improvement, etc.).

[0269] Step 3:

[0270] The terminal transmits the user's scenario selection information to the server.

[0271] Step 4:

[0272] The server reads data corresponding to the selected scenario from the database.

[0273] Step 5:

[0274] The server reviews the user's past practice history and customizes the scenario, which can include adjusting the practice level or focusing on specific tasks.

[0275] Step 6:

[0276] The server generates the AI ​​character's configuration data, which includes the character's appearance, personality, intonation, and facial expression patterns generated by the emotion engine.

[0277] Step 7:

[0278] The server sends the AI ​​character configuration data generated by the server to the device.

[0279] Step 8:

[0280] The device displays the AI ​​character to the user and prepares to begin a conversation.

[0281] Step 9:

[0282] The user initiates the interaction and speaks the first word (e.g., beginning a presentation).

[0283] Step 10:

[0284] The device records the user's voice input in real time and sends the data to the server.

[0285] Step 11:

[0286] The server receives the recorded data, performs voice analysis, and analyzes the user's linguistic intent.

[0287] Step 12:

[0288] The server uses an emotion engine to analyze the facial expression data collected at the same time as the user's speech data, and recognizes the user's emotions.

[0289] Step 13:

[0290] The server generates an appropriate response based on the user's intentions and emotions, and adjusts the character's facial expression and intonation.

[0291] Step 14:

[0292] The server sends the generated response data to the terminal.

[0293] Step 15:

[0294] The device changes the AI ​​character's facial expression and intonation to provide responses to the user in voice or text.

[0295] Step 16:

[0296] The user makes the next utterance and the dialogue proceeds. The device again records the user's voice input and sends it to the server.

[0297] Step 17:

[0298] The server analyzes the user's continuous speech data and facial expression data, and generates feedback in real time based on the user's emotions and speech content.

[0299] Step 18:

[0300] The server transmits the generated feedback data to the terminal.

[0301] Step 19:

[0302] The device provides real-time feedback to the user via voice or text, providing suggestions for improvement and advice.

[0303] Step 20:

[0304] The server stores the user's interaction history and feedback in a database.

[0305] Step 21:

[0306] The server analyzes the user's learning status based on the stored data and generates data to customize the content of the next practice session.

[0307] Step 22:

[0308] The device will then display a report of the results to the user, including a score for the exercise, specific areas for improvement, and advice for the next exercise.

[0309] Through these detailed steps, users can effectively improve their skills while experiencing realistic interactions. By utilizing the emotion engine, appropriate responses and feedback are provided that are tailored to the user's emotions, creating a more natural and effective learning environment.

[0310] Example 2

[0311] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0312] Conventional dialogue simulator systems lack real-time feedback on user voice input and dialogue content, making it difficult to customize specific scenarios that effectively reflect the user's emotions and practice history.Furthermore, it is difficult to provide a realistic dialogue environment by changing facial expressions and intonation during dialogue.

[0313] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and transmit character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and transmit it to the server; means for the server to analyze the user's utterance data, generate an appropriate response, and transmit it to the terminal; means for the terminal to respond while changing the AI ​​character's facial expression and intonation; means for the server to recognize the user's emotions using an emotion engine and adjust the response content and feedback based on the recognition; means for transmitting the feedback generated by the server to the terminal and providing it to the user; and means for the server to save the user's dialogue history and feedback and use it for future learning. This enables advanced scenario customization that reflects the user's practice history and emotions and feedback with real-time emotion recognition.

[0314] A "simulator system" is a computer-based system for conducting interactive training for a user.

[0315] A "user" is an individual who uses this system to select a scenario and conduct dialogue training.

[0316] A "server" is a computer that centrally processes the entire system, analyzes input data from users, and generates appropriate scenarios and responses.

[0317] A "terminal" is a device through which a user accesses the system and uses the functions of the dialogue simulator system.

[0318] A "scenario" refers to a specific situation or setting prepared for dialogue training.

[0319] A "database" is a system for managing and storing data used by the dialogue simulator system.

[0320] "Customization" refers to adjusting and changing scenarios and settings based on a user's past history and specific needs.

[0321] An "AI character" is a character with artificial intelligence that converses with the user within the dialogue simulator system.

[0322] "Configuration data" is information used to determine the appearance, tone of voice, and behavior of an AI character.

[0323] "Utterance data" is data of the voice uttered by the user during a conversation.

[0324] "Natural language processing" is a technology that enables computers to understand and generate human language.

[0325] An "emotion engine" is software that analyzes a user's voice and facial expressions to determine their emotional state.

[0326] "Feedback" refers to evaluations and advice given to a user's interactions and actions.

[0327] The "dialogue history" is a record of the content of dialogues and scenarios that the user has had in the past.

[0328] "Intonation" refers to the intonation and pitch of the voice when speaking.

[0329] "Facial expressions" are expressions on the user's face that indicate emotions and reactions.

[0330] The dialogue simulator system of the present invention is structured so that the user accesses the system and selects a scenario, and the entire system is processed by a server. The system analyzes the user's speech data, facial expression data, and dialogue history to provide an environment for advanced dialogue with the user.

[0331] The specific hardware required to implement this system includes the devices used by users (e.g., PCs, smartphones, tablets) and the server that controls the entire system. The main software used includes a speech recognition API (e.g., Google Cloud Speech-to-Text API), a natural language processing module, an emotion recognition API (e.g., Microsoft Azure Cognitive Services Emotion API), and a module for AI character generation.

[0332] When starting to use the system, the user accesses the dialogue simulator system through their own device and checks the scenario selection menu on the interface. When the user selects the desired scenario, the selection information is sent from the device to the server. The server reads data based on the selected scenario from the database and customizes the scenario by referring to the user's past practice history. Based on the customized scenario, the server generates configuration data for the AI ​​character and sends it to the device.

[0333] When a user initiates a dialogue, the device records the user's voice input in real time and sends it to the server. The server analyzes the voice data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends the response data to the device. The device responds to the user by changing the AI ​​character's facial expression and intonation.

[0334] In addition, the server uses an emotion engine to recognize the user's emotions and adjusts responses and feedback accordingly. For example, if the user is nervous, the AI ​​character will respond in an encouraging manner to ease the user's tension. The server also generates specific feedback based on the results of analyzing the user's speech and facial expression data and sends it to the device. The generated feedback is provided to the user in the form of voice or text, and includes specific advice and areas for improvement.

[0335] Furthermore, the server stores the user's dialogue history and feedback in a database. This stored data is used as personalized data for the next scenario customization, allowing the user to continuously receive dialogue training that is individually adapted to their needs.

[0336] As a concrete example, let's say a user selects the "Improve Presentation Skills" scenario and begins a presentation. When the user says, "Today I'm going to give a presentation about our new product," the device records this speech and sends it to the server. The server analyzes the audio data and generates a response such as, "Tell me more about your new product." At this time, it analyzes the user's emotions using an emotion engine and adjusts the tone and content of the response as necessary. After the presentation ends, the server analyzes the content and intonation of the user's speech and provides specific feedback such as, "The part of your presentation that was particularly easy to understand was the product explanation. However, it would be even better if you emphasized your intonation a little more."

[0337] An example of a prompt in this system is, "You have selected a scenario for improving your presentation skills and have begun your presentation. Please say, 'Today I will be presenting about our new product.'"

[0338] This allows the dialogue simulator system to help users improve their skills and provide a realistic practice environment.

[0339] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0340] Step 1:

[0341] The user accesses the dialogue simulator system and selects a scenario.

[0342] Specific operation: The user opens the browser on the device, accesses the system URL, and selects "Improve Presentation Skills" from the scenario menu.

[0343] Input: Scenario information selected by the user when they access the system.

[0344] Output: The selected scenario information is sent from the terminal to the server.

[0345] Step 2:

[0346] The server loads data corresponding to the selected scenario and customizes the scenario.

[0347] Specific operation: The server loads the "Improve Presentation Skills" scenario template from the database. It then refers to the user's past history and adjusts and modifies the scenario.

[0348] Input: Selected scenario information and user practice history data.

[0349] Output: Customized scenario data.

[0350] Step 3:

[0351] The server generates configuration data for the AI ​​character and sends it to the device.

[0352] Specific operation: Set the AI ​​character's name, appearance, tone of voice, etc., and send the setting data to the device in JSON format.

[0353] Input: Customized scenario data.

[0354] Output: AI character configuration data is sent to the device.

[0355] Step 4:

[0356] The user starts a dialogue, and the terminal receives the utterance data and transmits it to the server.

[0357] Specific operation: The user says, "Today I will be giving a presentation about our new product." The device records the voice and sends it to the server.

[0358] Input: User's voice input.

[0359] Output: The recorded audio data is sent to the server.

[0360] Step 5:

[0361] The server analyzes the user's voice data, generates an appropriate response, and sends it to the terminal.

[0362] Specific operation: The voice data is converted to text using the Google Cloud Speech-to-Text API, and the text data is analyzed using a natural language processing module to generate a response.

[0363] Input: Recorded audio data.

[0364] Output: Response data such as "Please tell me more about your new product" is generated and sent to the terminal.

[0365] Step 6:

[0366] The device responds by changing the AI ​​character's facial expression and intonation.

[0367] Specific operation: The text response is converted into speech using a speech synthesis API, and the response is played back as speech while the AI ​​character's facial expression changes.

[0368] Input: Response data.

[0369] Output: A voice-synthesized response and facial expression changes are provided to the user.

[0370] Step 7:

[0371] The server uses an emotion engine to recognize the user's emotions and adjusts responses and feedback accordingly.

[0372] Specific behavior: Analyzes emotions using the emotion engine (Microsoft Azure Cognitive Services Emotion API) and adjusts the tone and content of responses.

[0373] Input: User's voice and facial expression data.

[0374] Output: The adjusted response data.

[0375] Step 8:

[0376] The server generates feedback and sends it to the terminal for presentation to the user.

[0377] Specific actions: Analyze the user's speech content and intonation and generate specific feedback, such as, "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more."

[0378] Input: Speech content, intonation, and facial expression data.

[0379] Output: The generated feedback data is sent to the terminal and provided to the user.

[0380] Step 9:

[0381] The server stores the user's interaction history and feedback and uses it for future learning.

[0382] Specific operation: After the dialogue session ends, the spoken content and emotional data are saved in a database. The next time a scenario is customized, an individually adapted scenario is generated based on the saved data.

[0383] Input: User interaction history and feedback data.

[0384] Output: Stored interaction history and feedback data.

[0385] (Application example 2)

[0386] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0387] In today's brick-and-mortar stores, improving the customer service skills of store associates is important, but there is a lack of systems for effectively training them. Furthermore, traditional training methods lack consistency and objectivity, making it difficult to provide individualized instruction tailored to each employee's strengths and weaknesses.

[0388] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and send character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and send it to the server; means for the server to analyze the user's utterance data, generate an appropriate response, and send it to the terminal; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation and generate feedback; means for the server to send the generated feedback to the terminal and provide it to the user; means for the server to save the user's dialogue history and feedback and use it for future learning; means for providing customer service training scenarios to the terminal, including a function to perform emotion analysis in real time and adjust responses; and means for the terminal to present detailed feedback and areas for improvement after the training is completed. This enables consistent, high-quality customer service training and effectively supports employee skill improvement.

[0389] A "dialogue simulator system" is a system in which the user selects a scenario and engages in dialogue with an AI character based on that scenario.

[0390] A "scenario" is a set situation or theme for a user to practice with the dialogue simulator.

[0391] The "server" is a central processing unit that processes the scenarios and speech data selected by the user, generates AI characters, and provides feedback.

[0392] An "AI character" is a virtual character equipped with artificial intelligence that is generated in a dialogue simulator system to engage in dialogue with a user.

[0393] A "terminal" is a computer device that allows a user to access the dialogue simulator system and select a scenario or engage in dialogue.

[0394] "Utterance data" is voice data of what the user says in the dialogue simulator system.

[0395] "Feedback" is information indicating advice and areas for improvement provided based on the analysis of the user's dialogue content, facial expressions, and speaking style.

[0396] "Emotion analysis" is a technology that analyzes a user's emotional state from their speech and facial expressions.

[0397] A "customer service training scenario" is a specific interaction situation designed to help store staff improve their customer service skills.

[0398] "History" is a record of the interactions a user has had and feedback they have received in the past.

[0399] "Consistency" means that customer service training using the dialogue simulator system is always conducted with the same quality.

[0400] "High-quality customer service training" refers to providing effective training to improve customer service skills that is tailored to the strengths and weaknesses of each individual store employee.

[0401] The embodiment of the present invention is aimed at "customer service training" and specifically includes a step in which a user utilizes a dialogue simulator system to improve customer service skills.

[0402] First, a user accesses the dialogue simulator system using a device (e.g., a smartphone) and selects a specific scenario for training (e.g., introducing a new product or handling a complaint). The device then sends the selection information to the server, which then reads the scenario data from a database and customizes the scenario based on the user's past practice history.

[0403] Next, the server generates an AI character based on the customized scenario and sends the character's configuration data to the device. The device then prepares for the user to begin a dialogue, and the user speaks to simulate a real dialogue. For example, a user who selected a new product introduction scenario might say, "Tell me about the new products that have just arrived."

[0404] As the conversation progresses, the device sends the user's speech data in real time to the server, which then analyzes it using a natural language processing (NLP) engine. Specifically, it uses the Google Cloud Natural Language API to understand what the user is saying and generate appropriate responses based on that. This process also utilizes generative AI models.

[0405] In addition to generating responses, the server uses Affectiva's emotion recognition engine to analyze the user's emotional state and adjust the AI ​​character's facial expressions and intonation accordingly. The server also generates detailed feedback based on the analysis of the speech and facial expression data, suggesting areas for improvement to the user. This feedback includes specific advice and areas for improving effective customer service skills. After training is complete, the feedback is returned to the terminal and presented to the user.

[0406] The server stores this dialogue history and feedback in a database and uses it as data to make the next training session more effective.The next time a scenario is selected, it is automatically customized based on past history, providing the user with the optimal training environment.

[0407] For example, if a user says, "Today I'm going to give a presentation about our new product," the server will respond with, "Tell me more about your new product." If the user is nervous, the AI ​​character will offer words of encouragement, such as, "Relax and let's talk. Let's do our best together."

[0408] Examples of prompts:

[0409] "New product introduction"

[0410] User: "Tell me about the new items you've got in stock."

[0411] AI Character: "This new product uses the latest technology and offers 50% longer battery life than the previous model. Plus, it has a new, stylish design."

[0412] In this way, the dialogue simulator system effectively supports the improvement of customer service skills in brick-and-mortar stores, enabling store clerks to provide high-quality service in actual customer service situations.

[0413] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0414] Step 1:

[0415] A user accesses a terminal and starts the dialogue simulator system. A scenario selection menu is displayed so that the user can select a specific customer service training scenario.

[0416] Input: User's scenario selection

[0417] Output: Selected scenario information

[0418] Specific operation: The terminal acquires the scenario information selected by the user from the menu, formats the selected information, and sends it to the server.

[0419] Step 2:

[0420] Based on the scenario selection information received by the server, the server reads the corresponding scenario data from the database and customizes the scenario based on the user's past practice history.

[0421] Input: Selected scenario information, user's past practice history

[0422] Output: Customized scenario data

[0423] Specific operation: The server queries the database to obtain the user's practice history and selected scenario data. Based on this data, it customizes the scenario using scripts and a rules engine.

[0424] Step 3:

[0425] The server generates an AI character based on the customized scenario data and sends the character's setting data to the device.

[0426] Input: Customized scenario data

[0427] Output: AI character configuration data

[0428] Specific operation: The generative AI model is used to generate dialogue and facial expression data for the AI ​​character, and the setting data is sent to the device.

[0429] Step 4:

[0430] The user starts a conversation with the AI ​​character on their device. The user's speech is recorded by the device's microphone and sent to the server in real time.

[0431] Input: User utterance

[0432] Output: Sending speech data to the server

[0433] Specific operation: The device records the user's speech as audio data, converts it into an appropriate format, and sends it to the server.

[0434] Step 5:

[0435] The server analyzes the user's speech data using a natural language processing engine (Google Cloud Natural Language API) and generates an appropriate response.

[0436] Input: User utterance data

[0437] Output: The generated response

[0438] How it works: The server inputs the speech data into a natural language processing engine, analyzes the user's intent, and uses a generative AI model to generate an appropriate response.

[0439] Step 6:

[0440] The server analyzes the user's speech and facial expression data using an emotion analysis engine (Affectiva) and adjusts the AI ​​character's facial expressions and intonation.

[0441] Input: Speech data, facial expression data

[0442] Output: Adjusted response data

[0443] How it works: The server inputs speech data and facial expression data into an emotion analysis engine to identify the user's emotional state, and then adjusts the AI ​​character's facial expression and intonation accordingly.

[0444] Step 7:

[0445] The server sends the response and analysis results to the terminal, which displays them to the user.

[0446] Input: Adjusted response data

[0447] Output: The response that is displayed to the user

[0448] Specific operation: Based on the response data received from the server, the device updates the display of the AI ​​character and provides a response to the user along with audio.

[0449] Step 8:

[0450] After the training is completed, the server analyzes the user's interaction history and generates detailed feedback and suggestions for improvement.

[0451] Input: Dialogue history

[0452] Output: Detailed feedback

[0453] How it works: The server analyzes data from completed training sessions and uses a generative AI model to generate feedback, including analysis of the user's speech and facial expressions.

[0454] Step 9:

[0455] The server generates feedback and sends it to the terminal for presentation to the user.

[0456] Input: Detailed feedback

[0457] Output: Feedback presented to the user

[0458] Specific operation: The device acquires feedback data and provides it to the user in text or voice.

[0459] Step 10:

[0460] The server stores the interaction history and feedback in a database for use in future training sessions.

[0461] Input: Dialogue history, feedback

[0462] Output: Saved data

[0463] Specific operation: The server stores the dialogue history and feedback in a database, and refers to this the next time a scenario is selected to customize it.

[0464] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0465] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0466] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0467] [Second embodiment]

[0468] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0469] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0470] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0471] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0472] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0473] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0474] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0475] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0476] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0477] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0478] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0479] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0480] The dialogue simulator system of the present invention begins when a user accesses the dialogue simulator system and selects a scenario. Once the user selects a scenario, the terminal transmits the selected scenario information to the server. The server reads data corresponding to the selected scenario from the database and customizes the scenario based on the user's past practice history.

[0481] The server then generates an AI character, which is configured to interact with the user, and the configuration data includes the character's appearance, personality, intonation, and facial expression patterns. The generated configuration data is sent to the device, which then displays it to the user.

[0482] When a user initiates a conversation, the device records the user's speech in real time and sends it to the server. The server analyzes the speech data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends this response data to the device. The device then provides a response to the user through an AI character, whose facial expressions and intonation also change according to the situation.

[0483] The server then analyzes the user's speech, facial expressions, and intonation in real time and generates feedback based on these. The feedback is then sent to the device and provided to the user in voice or text format. This feedback includes specific advice and suggestions for improvement.

[0484] Finally, the server stores the user's interaction history and feedback in a database, which is used for future practice and to analyze the user's progress and trends to customize the next scenario.

[0485] Specific examples

[0486] For example, if a user selects a scenario for improving presentation skills, the device sends "Improve presentation skills" to the server, and the server loads and customizes the scenario based on the associated template and past practice data.

[0487] To start a presentation, the user says, "Today I'm going to present about our new product." The device records this speech and sends it to the server. The server analyzes this information, generates a response such as, "Tell me more about your new product," and sends it back to the user via the device.

[0488] While the user continues the presentation, the server analyzes the user's speech content and intonation and generates appropriate feedback in real time. For example, after the presentation, specific feedback such as "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more" is provided.

[0489] Through these steps, users can effectively improve their presentation skills. This system provides an environment that is as close as possible to real conversations, and supports users in improving their skills.

[0490] The processing flow will be explained below.

[0491] Step 1:

[0492] The user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface.

[0493] Step 2:

[0494] The user selects the scenario of their choice (sales role-play, meeting practice, presentation skills improvement, etc.).

[0495] Step 3:

[0496] The terminal transmits the user's scenario selection information to the server.

[0497] Step 4:

[0498] The server reads data corresponding to the selected scenario from the database.

[0499] Step 5:

[0500] The server checks the user's past practice history and customizes the scenario.

[0501] Step 6:

[0502] The server generates configuration data for the AI ​​character, including the character's appearance, personality, intonation, and facial expression patterns.

[0503] Step 7:

[0504] The server sends the AI ​​character configuration data generated by the server to the device.

[0505] Step 8:

[0506] The device displays the AI ​​character to the user, preparing the user to begin interaction.

[0507] Step 9:

[0508] The user initiates the interaction and speaks the first word (e.g., beginning a presentation).

[0509] Step 10:

[0510] The device records the user's voice input and transmits the data to the server in real time.

[0511] Step 11:

[0512] The server analyzes the voice data and uses natural language processing to understand the user's intent.

[0513] Step 12:

[0514] The server generates an appropriate response based on the user's intentions, taking into account the character's personality and the scenario content.

[0515] Step 13:

[0516] The server generates a response and sends it to the terminal.

[0517] Step 14:

[0518] The device responds to the user in voice or text, changing the AI ​​character's facial expression and intonation.

[0519] Step 15:

[0520] The user then makes the next utterance and the dialogue continues. The device again records the voice input and sends it to the server.

[0521] Step 16:

[0522] The server analyzes the user's speech in real time and generates feedback based on the speech content, facial expressions, and speaking intonation.

[0523] Step 17:

[0524] The server generates feedback and sends it to the device.

[0525] Step 18:

[0526] The terminal provides the generated feedback to the user in real time by voice or text.

[0527] Step 19:

[0528] The server stores the user's interaction history and feedback in a database.

[0529] Step 20:

[0530] The server analyzes the user's learning status based on the stored data and generates data to customize the content of the next practice session.

[0531] Step 21:

[0532] The device will then display a report of the results to the user, including a score, specific areas for improvement, and suggestions for the next practice session.

[0533] Through the above steps, users can effectively improve their skills through interaction.

[0534] Example 1

[0535] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0536] Current dialogue simulation systems suffer from monotonous reactions to user utterances and a lack of appropriate feedback that takes into account the user's past practice history. This makes it difficult for users to achieve sufficient learning results, particularly in finer details such as pronunciation, intonation, and facial expressions, which can slow down improvement. Furthermore, the lack of advanced conversational simulation using generative AI models limits the user experience.

[0537] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0538] In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for a terminal to transmit the selected scenario information to the server; means for the server to read scenario data and the user's past practice history from a database and customize the scenario; means for the server to generate an AI character and transmit character setting data to the terminal; means for the terminal to record utterance data for the user to start a dialogue in real time and transmit it to the server; means for the server to analyze the voice data, understand the user's intention using natural language processing and generate an appropriate response; means for the server to transmit the generated response data to the terminal and provide the response to the user through the AI ​​character; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation, generate feedback, and transmit the generated feedback to the terminal and provide it to the user; and means for the server to store the dialogue history and feedback in a database and use it for future learning. This allows users to practice in an environment that more closely resembles actual dialogue and receive detailed feedback to improve their skills.

[0539] A "user" is a subject who uses the dialogue simulator system to select a scenario and engage in dialogue.

[0540] The "dialogue simulator system" is a system that simulates dialogue based on a scenario selected by the user and provides reactions and feedback from an AI character.

[0541] "Scenario" refers to a practice or scene setting selected by the user in the dialogue simulator system.

[0542] A "terminal" is a hardware device (for example, a personal computer or smartphone) that allows a user to access and operate the dialogue simulator system.

[0543] The "server" is the central component of the dialogue simulator system, and is a computer that loads and customizes scenario data, generates AI characters, analyzes user speech data, and generates responses.

[0544] "Scenario data" refers to data such as templates, storylines, and example questions and answers related to each scenario in the dialogue simulator system.

[0545] The "database" is a data storage system for storing scenario data, the user's past practice history, dialogue history, feedback, and so on.

[0546] "Customization" is the process of adaptively modifying and adjusting a scenario, taking into account the user's past practice history and other relevant data.

[0547] An "AI character" is an artificial intelligence-based character (avatar) generated in a dialogue simulator system to engage in dialogue with a user.

[0548] "Character setting data" refers to data that includes information such as the AI ​​character's appearance, personality, intonation, and facial expression patterns.

[0549] "Utterance data" refers to recorded data of voice utterances made by a user during a conversation.

[0550] "Natural language processing" is a technology that allows an AI character to understand what the user is saying and generate an appropriate response based on that.

[0551] "Response data" is data that includes a response or reaction generated by the server in response to the content of the user's utterance.

[0552] "Feedback" is evaluation information that includes advice and suggestions for improving the dialogue, and is generated based on the user's speech content, facial expressions, intonation, etc.

[0553] "Dialogue history" refers to a record of the dialogues that a user has had with the dialogue simulator system.

[0554] The dialogue simulator system of the present invention begins when a user accesses the dialogue simulator system and selects a scenario. The user accesses the system from a browser on a PC or smartphone, logs in, and then selects a scenario that suits their purpose from a list of scenarios provided. For example, the user can select the scenario "Improve Presentation Skills."

[0555] Once a selection is made, the device sends this information to the server via an HTTP request (for example, the POST method). At this time, the data sent includes the user ID and the selected scenario ID. On the server side, the corresponding scenario data is read from a database (for example, MySQL or PostgreSQL) based on the received scenario ID. At the same time, the user's past practice history is obtained and the scenario is customized based on this. This customization reflects the themes the user has previously experienced and the points that should be emphasized.

[0556] Next, the server generates an AI character. This AI character is generated in real time using, for example, a custom Unity API. The generated character's appearance, personality, intonation, facial expression patterns, etc. are configured. This configuration data is sent to the device in JSON format, and the device uses WebGL or WebRTC technology to display it on the screen.

[0557] When a conversation begins, the device records the user's speech in real time and performs noise cancellation and echo removal to obtain high-quality audio data. This audio data is sent to a server, where it is converted into text using speech recognition technology such as the Google Cloud Speech-to-Text API. The server then analyzes the user's intent using natural language processing with a generative AI model (e.g., BERT or GPT-3). Based on the results of this analysis, the server generates an appropriate response and sends the response data back to the device.

[0558] Based on the received response data, the device uses voice synthesis technology such as the Web Speech API to have the AI ​​character respond, changing facial expressions and intonation along with the response to achieve a more natural dialogue.

[0559] Furthermore, the server analyzes the user's speech content, facial expressions, and speaking intonation in real time, and provides specific feedback based on the results. This feedback may include detailed advice such as, "The product explanation section was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more." This feedback is sent from the server to the device, which then provides it to the user. This process is carried out using emotion analysis tools such as Microsoft Azure's Emotion API.

[0560] Finally, the server stores all interaction history and feedback in a database, which can be used for future practice sessions, to analyze the user's progress, and to provide more customized scenarios.

[0561] Examples of specific examples and prompts

[0562] For example, if a user selects the "Improve Presentation Skills" scenario, the scenario will include the following prompt:

[0563] "Please tell us three features of your new product."

[0564] "Please suggest appropriate responses to the audience questions."

[0565] "Please tell me how to effectively summarize at the end of a presentation."

[0566] When a user says, "Today I'm going to give a presentation about our new product," the device's microphone records this speech and sends it to the server, which then responds with, "Please tell us more about your new product." After the presentation, the server analyzes the user's speech and provides feedback such as, "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more."

[0567] This system allows users to improve even the finer details of pronunciation, intonation, and facial expressions through dialogue.

[0568] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0569] Step 1:

[0570] This is the procedure in which a user accesses the dialogue simulator system and selects a scenario.

[0571] Input: User login information, system access, scenario selection request.

[0572] Data processing and calculation: User authentication, scenario list display, confirmation of selected scenario.

[0573] Output: Selected scenario ID, user ID.

[0574] Specific operation: A user logs in from a PC or smartphone browser, browses the list of scenarios, and then selects a scenario such as "Improve Presentation Skills."

[0575] Step 2:

[0576] This is a procedure in which the terminal transmits scenario selection information to the server.

[0577] Input: Selected scenario ID, user ID.

[0578] Data processing and calculation: Generating and sending HTTP requests (POST method, etc.).

[0579] Output: The scenario selection request sent to the server.

[0580] Specific operation: The device generates an HTTP request including the selected scenario ID and user ID and sends it to the server.

[0581] Step 3:

[0582] This is the procedure in which the server reads scenario data and the user's past practice history from the database and customizes the scenario.

[0583] Input: Scenario ID, User ID.

[0584] Data processing and calculation: Performing database queries, filtering and extracting past exercise history, and adaptive scenario customization.

[0585] Output: Customized scenario data.

[0586] Specific operation: The server reads the relevant scenario data and the user's past practice history from the MySQL database, and customizes the scenario based on the previous practice content.

[0587] Step 4:

[0588] This is the procedure in which the server generates an AI character and sends the character's setting data to the terminal.

[0589] Input: Customized scenario data.

[0590] Data processing and calculation: Calling the AI ​​character generation module and generating setting data.

[0591] Output: AI character configuration data (JSON format).

[0592] How it works: The server uses Unity's custom API to generate a character, create settings such as appearance, personality, intonation, and facial expression patterns, and send them to the device.

[0593] Step 5:

[0594] This is the procedure in which the device displays an AI character and begins a dialogue with the user.

[0595] Input: AI character configuration data.

[0596] Data processing and calculation: Reading setting data, using the screen drawing library.

[0597] Output: An AI character displayed on the screen.

[0598] Specific operation: The device displays an AI character on the screen based on the configuration data received using WebGL and WebRTC technology, and the conversation begins when the user presses the "Start" button.

[0599] Step 6:

[0600] This is a procedure in which the terminal records the user's speech data and transmits it to the server.

[0601] Input: User utterance.

[0602] Data processing and calculation: voice recording, noise cancellation, echo removal.

[0603] Output: Preprocessed audio data.

[0604] Specific operation: When a user speaks to an AI character, the device's microphone records the voice, performs noise cancellation and echo removal in real time, and then sends the voice data to the server.

[0605] Step 7:

[0606] This is the procedure where the server analyzes the voice data and generates an appropriate response.

[0607] Input: Preprocessed audio data.

[0608] Data processing and calculation: Speech-to-Text, natural language processing (BERT and GPT-3).

[0609] Output: Response data (text format).

[0610] How it works: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text, then uses a natural language processing model to analyze the user's intent and generate an appropriate response.

[0611] Step 8:

[0612] This is the procedure by which the device displays a response to the user through an AI character.

[0613] Input: Response data from the server (text format).

[0614] Data processing and calculation: voice synthesis, facial animation generation.

[0615] Output: Voice responses and facial expressions by the AI ​​character.

[0616] How it works: The device uses the Web Speech API to convert the response into voice and responds to the user along with the AI ​​character's facial expressions, resulting in a more natural dialogue.

[0617] Step 9:

[0618] This is a procedure in which the server analyzes the user's speech and generates real-time feedback.

[0619] Input: User utterance and parsed results.

[0620] Data processing and calculation: Sentiment analysis, speech evaluation, and extraction of areas for improvement.

[0621] Output: Feedback data.

[0622] Specific operation: The server uses Microsoft Azure's Emotion API and emotion analysis tools to evaluate the user's speech and generate feedback including specific improvements and advice.

[0623] Step 10:

[0624] This is a procedure in which the terminal provides feedback to the user.

[0625] Input: Feedback data.

[0626] Data processing and calculation: Display of feedback contents.

[0627] Output: The displayed feedback.

[0628] Specific operation: The device provides the received feedback data to the user by voice or text, using a text box displayed on the screen or a speaker for voice feedback.

[0629] Step 11:

[0630] This is the procedure by which the server stores the dialogue history and feedback in a database.

[0631] Input: Dialogue history, feedback data.

[0632] Data processing and calculation: Writing to database, saving data.

[0633] Output: Stored interaction history and feedback.

[0634] What it does: The server stores all interaction history and feedback in a MySQL database, which is used to customize the scenario for the next session.

[0635] (Application example 1)

[0636] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0637] Previously, training to improve customer service skills in brick-and-mortar stores required effort and cost to deal with real customers, and had the problem of not being able to receive immediate feedback. Furthermore, there was a lack of opportunities for new staff to practice interacting with real customers, making it difficult for them to improve their skills. For this reason, there is a need for a system that can improve customer service skills while receiving effective, real-time feedback.

[0638] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0639] In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and send character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and send it to the server; means for the server to analyze the user's utterance data to generate an appropriate response and send it to the terminal; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation and generate feedback; means for the server to send the generated feedback to the terminal and provide it to the user; means for the server to save the user's dialogue history and feedback and use it for future learning; means for the terminal to function as a customer service simulator and enable brick-and-mortar store staff to improve their customer service skills; and means for providing interactive feedback to the user. This enables brick-and-mortar store staff to improve their customer service skills by receiving feedback effectively and in real time.

[0640] A "dialogue simulator system" is a system that allows users to improve specific skills or knowledge by engaging in dialogue in a simulated format.

[0641] A "scenario" is a set of stories or conditions designed to simulate a particular situation or dialogue.

[0642] A "server" is a computer system that serves as the core of the dialogue simulator system accessed by users, and is a device that processes data and generates AI characters.

[0643] "Means for reading data and customizing a scenario" refers to a mechanism in which the server reads related information from a database for a scenario selected by a user and modifies it to suit the user's needs.

[0644] An "AI character" is a virtual character with artificial intelligence that is generated in a dialogue simulator to have an interactive conversation with a user.

[0645] "Configuration data" is data that defines the specific attributes and behavior of an AI character, such as their appearance, personality, intonation, and facial expression patterns.

[0646] "Utterance data" is data of the voice uttered by the user during a conversation.

[0647] A "reply" is a message that the server generates in response to an analysis of the user's speech data.

[0648] "Expressions" refer to the facial movements and emotional expressions shown by AI characters in the dialogue simulator.

[0649] "Intonation" refers to the rise and fall of the voice when speaking.

[0650] "Feedback" refers to information such as evaluations and areas for improvement provided based on the user's interactions and actions.

[0651] "Dialogue history" is a record of past dialogues between the user and the AI ​​character.

[0652] The "customer service simulator" is a system that allows store staff to simulate interactions with customers and improve their response skills.

[0653] "Interactive feedback" refers to dynamic evaluations and advice provided in real time in response to a user's actions.

[0654] This invention relates to a customer interaction simulator system that allows store staff to improve their customer interaction skills. The system starts when a user accesses the interaction simulator system and selects a scenario. Based on the selected scenario, the server loads data and customizes the scenario. The server then generates an AI character and sends its setting data to the terminal.

[0655] After the user initiates a dialogue, the device receives the user's speech data and sends it to the server. The server analyzes the user's speech data, generates an appropriate response using a generative AI model, and sends this response data to the device. The device then provides a response to the user through an AI character. At this time, the AI ​​character's facial expression and intonation also change according to the scenario. The server then analyzes the content of the user's speech, facial expression, and speaking intonation to generate feedback. The generated feedback is sent to the device and provided to the user.

[0656] Hardware and software used

[0657] Hardware:

[0658] Smart glasses (e.g., smart devices)

[0659] Head-mounted displays (e.g., mixed reality devices)

[0660] software:

[0661] Python (programming language)

[0662] speech_recognition (speech recognition library)

[0663] transformers (natural language processing library)

[0664] The server captures the user's voice input data and converts it into text using a speech recognition library. The converted data is then used by an AI character to generate an appropriate response using a natural language processing library. This response is then provided to the user via the device. The user's conversation is analyzed in real time and feedback is generated. This feedback includes specific advice and suggestions for improvement and is provided to the user interactively via the device.

[0665] Specific examples

[0666] For example, suppose a staff member at a brick-and-mortar store selects the "Basic Customer Response Scenario" and starts training. The staff member says, "Hello, is there anything you'd like to buy in our store?" The device records this speech, and the server analyzes the audio data. Based on the analysis results, the server generates a response such as "Hello, is there anything in particular you'd like to buy today?" and displays the response through the device.

[0667] The system then analyzes the speech and intonation of the staff member and provides feedback such as, "If you ask your question with a more emphasized intonation, it will be easier for the customer to understand." This feedback is provided interactively in real time, helping the staff member improve their skills.

[0668] In this way, store staff can effectively improve their customer service skills.

[0669] Example prompt sentence:

[0670] "Hello, are you looking for something in the store?"

[0671] "Is there anything in particular you're looking for today?"

[0672] "If you ask questions with a bit more emphasis on intonation, you'll be able to communicate more clearly to the customer."

[0673] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0674] Step 1:

[0675] A user accesses the dialogue simulator system and selects a scenario. As input, the user accesses the scenario selection interface from the device they use (smart glasses or a head-mounted display) and selects a specific scenario (e.g., "Basic Customer Interaction Scenario"). As output, the selected scenario information is sent to the server.

[0676] Step 2:

[0677] The server reads the data corresponding to the selected scenario and customizes the scenario. The input is the scenario selection information sent by the user, and based on this the server retrieves related information from the database. The data is processed by customizing the scenario based on the user's past practice history, and customized scenario data is generated as output.

[0678] Step 3:

[0679] The server generates an AI character and sends the character's configuration data to the device. The input is customized scenario data, and the AI ​​character's configuration (appearance, personality, intonation, facial expression patterns, etc.) is generated. The data is processed using a generative AI model to generate the character, and the configuration data is generated as output and sent to the device.

[0680] Step 4:

[0681] The user starts a conversation. The device receives the user's speech data and sends it to the server. The input is the user's voice data at the start of the conversation, which the device captures in real time. The data is then processed using a speech recognition library to convert the voice data into text data, and the converted text data is generated as output and sent to the server.

[0682] Step 5:

[0683] The server analyzes the user's speech data, generates an appropriate response, and sends it to the device. The converted text data is used as input, and a response is generated using a generative AI model based on this. Natural language processing is used to process the data, analyzing the user's intent, and an appropriate response data is generated as output and sent to the device.

[0684] Step 6:

[0685] The device provides the user with a response from an AI character. The response data sent from the server is input, and the device receives this. The data is processed by changing the AI ​​character's facial expression and intonation based on the settings, and the response is provided to the user visually and audibly as output.

[0686] Step 7:

[0687] The server analyzes the user's speech content, facial expressions, and speaking intonation to generate feedback. The input is the user's latest speech data and dialogue history, and data calculations are performed using an AI model based on this data. The output is feedback that includes specific advice and suggestions for improvement.

[0688] Step 8:

[0689] The server sends the generated feedback to the terminal and provides it to the user. The generated feedback data is input and sent to the terminal. The data is processed by adjusting the display format of the feedback, and the feedback is provided to the user in an interactive form as output.

[0690] Step 9:

[0691] The server stores the user's interaction history and feedback and uses it for future learning. The input is the latest interaction history and feedback data, and the data is processed and stored appropriately in a database. The output is used to customize future scenarios and improve the learning algorithm.

[0692] Example prompt sentence:

[0693] "Hello, are you looking for something in the store?"

[0694] "Is there anything in particular you're looking for today?"

[0695] "If you ask questions with a bit more emphasis on intonation, you'll be able to communicate more clearly to the customer."

[0696] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0697] The dialogue simulator system of the present invention is structured so that a user accesses the system and selects a scenario, and the entire system is processed mainly by the server. The specific flow of processing will be explained below.

[0698] When starting to use the system, the user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface. The user selects the desired scenario (e.g., sales role-play, meeting practice, presentation skill improvement, etc.), and the selection information is sent from the terminal to the server.

[0699] The server reads data corresponding to the selected scenario from the database and customizes the scenario based on the user's past practice history. Based on this customized scenario, the server generates configuration data for the AI ​​character and sends the configuration data to the device. The device displays the generated AI character to the user, ready for the user to begin interacting with it.

[0700] When a user initiates a dialogue and speaks, the device records the user's voice input and sends it to the server in real time. The server analyzes the voice data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends the response data to the device. The device responds to the user by changing the AI ​​character's facial expression and intonation.

[0701] Furthermore, the system uses an emotion engine to analyze the user's speech and facial expression data. The server uses the emotion engine to recognize the user's emotions and adjusts the response and feedback accordingly. For example, if the user is nervous, the AI ​​character will respond in an encouraging manner to calm the user.

[0702] The server also uses the emotion engine to generate feedback. It analyzes the user's speech content, facial expressions, and speaking intonation in real time, and generates specific feedback based on the analysis results. This feedback is sent to the device and provided to the user in the form of voice or text. The feedback includes specific advice and areas for improvement.

[0703] For example, suppose a user selects a scenario for improving presentation skills and begins a presentation. When the user says, "Today I'm going to give a presentation about our new product," the device records this speech and sends it to the server. The server analyzes this speech data and generates a response such as, "Tell me more about your new product." It then analyzes the user's emotions using an emotion engine and adjusts the tone and content of the response as needed.

[0704] After the presentation, the server analyzes the user's speech and intonation, and provides specific feedback such as, "The product explanation was particularly easy to understand in the presentation. However, it would be even better if you emphasized the intonation a little more." The feedback is also adjusted appropriately, taking into account the user's emotions.

[0705] Finally, the server stores the user's dialogue history and feedback in a database and uses it as data to personalize the next scenario. In this way, the dialogue simulator system of the present invention helps users improve their skills and provides a more effective training environment.

[0706] To implement the invention, a program is required to smoothly carry out a series of operations, including user selection, data analysis, emotion recognition, and feedback generation, allowing users to effectively improve their practical skills through realistic interactions.

[0707] The processing flow will be explained below.

[0708] Step 1:

[0709] The user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface.

[0710] Step 2:

[0711] The user selects the scenario of their choice (e.g., sales role-play, meeting practice, presentation skill improvement, etc.).

[0712] Step 3:

[0713] The terminal transmits the user's scenario selection information to the server.

[0714] Step 4:

[0715] The server reads data corresponding to the selected scenario from the database.

[0716] Step 5:

[0717] The server reviews the user's past practice history and customizes the scenario, which can include adjusting the practice level or focusing on specific tasks.

[0718] Step 6:

[0719] The server generates the AI ​​character's configuration data, which includes the character's appearance, personality, intonation, and facial expression patterns generated by the emotion engine.

[0720] Step 7:

[0721] The server sends the AI ​​character configuration data generated by the server to the device.

[0722] Step 8:

[0723] The device displays the AI ​​character to the user and prepares to begin a conversation.

[0724] Step 9:

[0725] The user initiates the interaction and speaks the first word (e.g., beginning a presentation).

[0726] Step 10:

[0727] The device records the user's voice input in real time and sends the data to the server.

[0728] Step 11:

[0729] The server receives the recorded data, performs voice analysis, and analyzes the user's linguistic intent.

[0730] Step 12:

[0731] The server uses an emotion engine to analyze the facial expression data collected at the same time as the user's speech data, and recognizes the user's emotions.

[0732] Step 13:

[0733] The server generates an appropriate response based on the user's intentions and emotions, and adjusts the character's facial expression and intonation.

[0734] Step 14:

[0735] The server sends the generated response data to the terminal.

[0736] Step 15:

[0737] The device changes the AI ​​character's facial expression and intonation to provide responses to the user in voice or text.

[0738] Step 16:

[0739] The user makes the next utterance and the dialogue proceeds. The device again records the user's voice input and sends it to the server.

[0740] Step 17:

[0741] The server analyzes the user's continuous speech data and facial expression data, and generates feedback in real time based on the user's emotions and speech content.

[0742] Step 18:

[0743] The server transmits the generated feedback data to the terminal.

[0744] Step 19:

[0745] The device provides real-time feedback to the user via voice or text, providing suggestions for improvement and advice.

[0746] Step 20:

[0747] The server stores the user's interaction history and feedback in a database.

[0748] Step 21:

[0749] The server analyzes the user's learning status based on the stored data and generates data to customize the content of the next practice session.

[0750] Step 22:

[0751] The device will then display a report of the results to the user, including a score for the exercise, specific areas for improvement, and advice for the next exercise.

[0752] Through these detailed steps, users can effectively improve their skills while experiencing realistic interactions. By utilizing the emotion engine, appropriate responses and feedback are provided that are tailored to the user's emotions, creating a more natural and effective learning environment.

[0753] Example 2

[0754] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0755] Conventional dialogue simulator systems lack real-time feedback on user voice input and dialogue content, making it difficult to customize specific scenarios that effectively reflect the user's emotions and practice history.Furthermore, it is difficult to provide a realistic dialogue environment by changing facial expressions and intonation during dialogue.

[0756] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and transmit character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and transmit it to the server; means for the server to analyze the user's utterance data, generate an appropriate response, and transmit it to the terminal; means for the terminal to respond while changing the AI ​​character's facial expression and intonation; means for the server to recognize the user's emotions using an emotion engine and adjust the response content and feedback based on the recognition; means for transmitting the feedback generated by the server to the terminal and providing it to the user; and means for the server to save the user's dialogue history and feedback and use it for future learning. This enables advanced scenario customization that reflects the user's practice history and emotions and feedback with real-time emotion recognition.

[0757] A "simulator system" is a computer-based system for conducting interactive training for a user.

[0758] A "user" is an individual who uses this system to select a scenario and conduct dialogue training.

[0759] A "server" is a computer that centrally processes the entire system, analyzes input data from users, and generates appropriate scenarios and responses.

[0760] A "terminal" is a device through which a user accesses the system and uses the functions of the dialogue simulator system.

[0761] A "scenario" refers to a specific situation or setting prepared for dialogue training.

[0762] A "database" is a system for managing and storing data used by the dialogue simulator system.

[0763] "Customization" refers to adjusting and changing scenarios and settings based on a user's past history and specific needs.

[0764] An "AI character" is a character with artificial intelligence that converses with the user within the dialogue simulator system.

[0765] "Configuration data" is information used to determine the appearance, tone of voice, and behavior of an AI character.

[0766] "Utterance data" is data of the voice uttered by the user during a conversation.

[0767] "Natural language processing" is a technology that enables computers to understand and generate human language.

[0768] An "emotion engine" is software that analyzes a user's voice and facial expressions to determine their emotional state.

[0769] "Feedback" refers to evaluations and advice given to a user's interactions and actions.

[0770] The "dialogue history" is a record of the content of dialogues and scenarios that the user has had in the past.

[0771] "Intonation" refers to the intonation and pitch of the voice when speaking.

[0772] "Facial expressions" are expressions on the user's face that indicate emotions and reactions.

[0773] The dialogue simulator system of the present invention is structured so that the user accesses the system and selects a scenario, and the entire system is processed by a server. The system analyzes the user's speech data, facial expression data, and dialogue history to provide an environment for advanced dialogue with the user.

[0774] The specific hardware required to implement this system includes the devices used by users (e.g., PCs, smartphones, tablets) and the server that controls the entire system. The main software used includes a speech recognition API (e.g., Google Cloud Speech-to-Text API), a natural language processing module, an emotion recognition API (e.g., Microsoft Azure Cognitive Services Emotion API), and a module for AI character generation.

[0775] When starting to use the system, the user accesses the dialogue simulator system through their own device and checks the scenario selection menu on the interface. When the user selects the desired scenario, the selection information is sent from the device to the server. The server reads data based on the selected scenario from the database and customizes the scenario by referring to the user's past practice history. Based on the customized scenario, the server generates configuration data for the AI ​​character and sends it to the device.

[0776] When a user initiates a dialogue, the device records the user's voice input in real time and sends it to the server. The server analyzes the voice data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends the response data to the device. The device responds to the user by changing the AI ​​character's facial expression and intonation.

[0777] In addition, the server uses an emotion engine to recognize the user's emotions and adjusts responses and feedback accordingly. For example, if the user is nervous, the AI ​​character will respond in an encouraging manner to ease the user's tension. The server also generates specific feedback based on the results of analyzing the user's speech and facial expression data and sends it to the device. The generated feedback is provided to the user in the form of voice or text, and includes specific advice and areas for improvement.

[0778] Furthermore, the server stores the user's dialogue history and feedback in a database. This stored data is used as personalized data for the next scenario customization, allowing the user to continuously receive dialogue training that is individually adapted to their needs.

[0779] As a concrete example, let's say a user selects the "Improve Presentation Skills" scenario and begins a presentation. When the user says, "Today I'm going to give a presentation about our new product," the device records this speech and sends it to the server. The server analyzes the audio data and generates a response such as, "Tell me more about your new product." At this time, it analyzes the user's emotions using an emotion engine and adjusts the tone and content of the response as necessary. After the presentation ends, the server analyzes the content and intonation of the user's speech and provides specific feedback such as, "The part of your presentation that was particularly easy to understand was the product explanation. However, it would be even better if you emphasized your intonation a little more."

[0780] An example of a prompt in this system is, "You have selected a scenario for improving your presentation skills and have begun your presentation. Please say, 'Today I will be presenting about our new product.'"

[0781] This allows the dialogue simulator system to help users improve their skills and provide a realistic practice environment.

[0782] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0783] Step 1:

[0784] The user accesses the dialogue simulator system and selects a scenario.

[0785] Specific operation: The user opens the browser on the device, accesses the system URL, and selects "Improve Presentation Skills" from the scenario menu.

[0786] Input: Scenario information selected by the user when they access the system.

[0787] Output: The selected scenario information is sent from the terminal to the server.

[0788] Step 2:

[0789] The server loads data corresponding to the selected scenario and customizes the scenario.

[0790] Specific operation: The server loads the "Improve Presentation Skills" scenario template from the database. It then refers to the user's past history and adjusts and modifies the scenario.

[0791] Input: Selected scenario information and user practice history data.

[0792] Output: Customized scenario data.

[0793] Step 3:

[0794] The server generates configuration data for the AI ​​character and sends it to the device.

[0795] Specific operation: Set the AI ​​character's name, appearance, tone of voice, etc., and send the setting data to the device in JSON format.

[0796] Input: Customized scenario data.

[0797] Output: AI character configuration data is sent to the device.

[0798] Step 4:

[0799] The user starts a dialogue, and the terminal receives the utterance data and transmits it to the server.

[0800] Specific operation: The user says, "Today I will be giving a presentation about our new product." The device records the voice and sends it to the server.

[0801] Input: User's voice input.

[0802] Output: The recorded audio data is sent to the server.

[0803] Step 5:

[0804] The server analyzes the user's voice data, generates an appropriate response, and sends it to the terminal.

[0805] Specific operation: The voice data is converted to text using the Google Cloud Speech-to-Text API, and the text data is analyzed using a natural language processing module to generate a response.

[0806] Input: Recorded audio data.

[0807] Output: Response data such as "Please tell me more about your new product" is generated and sent to the terminal.

[0808] Step 6:

[0809] The device responds by changing the AI ​​character's facial expression and intonation.

[0810] Specific operation: The text response is converted into speech using a speech synthesis API, and the response is played back as speech while the AI ​​character's facial expression changes.

[0811] Input: Response data.

[0812] Output: A voice-synthesized response and facial expression changes are provided to the user.

[0813] Step 7:

[0814] The server uses an emotion engine to recognize the user's emotions and adjusts responses and feedback accordingly.

[0815] Specific behavior: Analyzes emotions using the emotion engine (Microsoft Azure Cognitive Services Emotion API) and adjusts the tone and content of responses.

[0816] Input: User's voice and facial expression data.

[0817] Output: The adjusted response data.

[0818] Step 8:

[0819] The server generates feedback and sends it to the terminal for presentation to the user.

[0820] Specific actions: Analyze the user's speech content and intonation and generate specific feedback, such as, "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more."

[0821] Input: Speech content, intonation, and facial expression data.

[0822] Output: The generated feedback data is sent to the terminal and provided to the user.

[0823] Step 9:

[0824] The server stores the user's interaction history and feedback and uses it for future learning.

[0825] Specific operation: After the dialogue session ends, the spoken content and emotional data are saved in a database. The next time a scenario is customized, an individually adapted scenario is generated based on the saved data.

[0826] Input: User interaction history and feedback data.

[0827] Output: Stored interaction history and feedback data.

[0828] (Application example 2)

[0829] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0830] In today's brick-and-mortar stores, improving the customer service skills of store associates is important, but there is a lack of systems for effectively training them. Furthermore, traditional training methods lack consistency and objectivity, making it difficult to provide individualized instruction tailored to each employee's strengths and weaknesses.

[0831] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and send character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and send it to the server; means for the server to analyze the user's utterance data, generate an appropriate response, and send it to the terminal; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation and generate feedback; means for the server to send the generated feedback to the terminal and provide it to the user; means for the server to save the user's dialogue history and feedback and use it for future learning; means for providing customer service training scenarios to the terminal, including a function to perform emotion analysis in real time and adjust responses; and means for the terminal to present detailed feedback and areas for improvement after the training is completed. This enables consistent, high-quality customer service training and effectively supports employee skill improvement.

[0832] A "dialogue simulator system" is a system in which the user selects a scenario and engages in dialogue with an AI character based on that scenario.

[0833] A "scenario" is a set situation or theme for a user to practice with the dialogue simulator.

[0834] The "server" is a central processing unit that processes the scenarios and speech data selected by the user, generates AI characters, and provides feedback.

[0835] An "AI character" is a virtual character equipped with artificial intelligence that is generated in a dialogue simulator system to engage in dialogue with a user.

[0836] A "terminal" is a computer device that allows a user to access the dialogue simulator system and select a scenario or engage in dialogue.

[0837] "Utterance data" is voice data of what the user says in the dialogue simulator system.

[0838] "Feedback" is information indicating advice and areas for improvement provided based on the analysis of the user's dialogue content, facial expressions, and speaking style.

[0839] "Emotion analysis" is a technology that analyzes a user's emotional state from their speech and facial expressions.

[0840] A "customer service training scenario" is a specific interaction situation designed to help store staff improve their customer service skills.

[0841] "History" is a record of the interactions a user has had and feedback they have received in the past.

[0842] "Consistency" means that customer service training using the dialogue simulator system is always conducted with the same quality.

[0843] "High-quality customer service training" refers to providing effective training to improve customer service skills that is tailored to the strengths and weaknesses of each individual store employee.

[0844] The embodiment of the present invention is aimed at "customer service training" and specifically includes a step in which a user utilizes a dialogue simulator system to improve customer service skills.

[0845] First, a user accesses the dialogue simulator system using a device (e.g., a smartphone) and selects a specific scenario for training (e.g., introducing a new product or handling a complaint). The device then sends the selection information to the server, which then reads the scenario data from a database and customizes the scenario based on the user's past practice history.

[0846] Next, the server generates an AI character based on the customized scenario and sends the character's configuration data to the device. The device then prepares for the user to begin a dialogue, and the user speaks to simulate a real dialogue. For example, a user who selected a new product introduction scenario might say, "Tell me about the new products that have just arrived."

[0847] As the conversation progresses, the device sends the user's speech data in real time to the server, which then analyzes it using a natural language processing (NLP) engine. Specifically, it uses the Google Cloud Natural Language API to understand what the user is saying and generate appropriate responses based on that. This process also utilizes generative AI models.

[0848] In addition to generating responses, the server uses Affectiva's emotion recognition engine to analyze the user's emotional state and adjust the AI ​​character's facial expressions and intonation accordingly. The server also generates detailed feedback based on the analysis of the speech and facial expression data, suggesting areas for improvement to the user. This feedback includes specific advice and areas for improving effective customer service skills. After training is complete, the feedback is returned to the terminal and presented to the user.

[0849] The server stores this dialogue history and feedback in a database and uses it as data to make the next training session more effective.The next time a scenario is selected, it is automatically customized based on past history, providing the user with the optimal training environment.

[0850] For example, if a user says, "Today I'm going to give a presentation about our new product," the server will respond with, "Tell me more about your new product." If the user is nervous, the AI ​​character will offer words of encouragement, such as, "Relax and let's talk. Let's do our best together."

[0851] Examples of prompts:

[0852] "New product introduction"

[0853] User: "Tell me about the new items you've got in stock."

[0854] AI Character: "This new product uses the latest technology and offers 50% longer battery life than the previous model. Plus, it has a new, stylish design."

[0855] In this way, the dialogue simulator system effectively supports the improvement of customer service skills in brick-and-mortar stores, enabling store clerks to provide high-quality service in actual customer service situations.

[0856] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0857] Step 1:

[0858] A user accesses a terminal and starts the dialogue simulator system. A scenario selection menu is displayed so that the user can select a specific customer service training scenario.

[0859] Input: User's scenario selection

[0860] Output: Selected scenario information

[0861] Specific operation: The terminal acquires the scenario information selected by the user from the menu, formats the selected information, and sends it to the server.

[0862] Step 2:

[0863] Based on the scenario selection information received by the server, the server reads the corresponding scenario data from the database and customizes the scenario based on the user's past practice history.

[0864] Input: Selected scenario information, user's past practice history

[0865] Output: Customized scenario data

[0866] Specific operation: The server queries the database to obtain the user's practice history and selected scenario data. Based on this data, it customizes the scenario using scripts and a rules engine.

[0867] Step 3:

[0868] The server generates an AI character based on the customized scenario data and sends the character's setting data to the device.

[0869] Input: Customized scenario data

[0870] Output: AI character configuration data

[0871] Specific operation: The generative AI model is used to generate dialogue and facial expression data for the AI ​​character, and the setting data is sent to the device.

[0872] Step 4:

[0873] The user starts a conversation with the AI ​​character on their device. The user's speech is recorded by the device's microphone and sent to the server in real time.

[0874] Input: User utterance

[0875] Output: Sending speech data to the server

[0876] Specific operation: The device records the user's speech as audio data, converts it into an appropriate format, and sends it to the server.

[0877] Step 5:

[0878] The server analyzes the user's speech data using a natural language processing engine (Google Cloud Natural Language API) and generates an appropriate response.

[0879] Input: User utterance data

[0880] Output: The generated response

[0881] How it works: The server inputs the speech data into a natural language processing engine, analyzes the user's intent, and uses a generative AI model to generate an appropriate response.

[0882] Step 6:

[0883] The server analyzes the user's speech and facial expression data using an emotion analysis engine (Affectiva) and adjusts the AI ​​character's facial expressions and intonation.

[0884] Input: Speech data, facial expression data

[0885] Output: Adjusted response data

[0886] How it works: The server inputs speech data and facial expression data into an emotion analysis engine to identify the user's emotional state, and then adjusts the AI ​​character's facial expression and intonation accordingly.

[0887] Step 7:

[0888] The server sends the response and analysis results to the terminal, which displays them to the user.

[0889] Input: Adjusted response data

[0890] Output: The response that is displayed to the user

[0891] Specific operation: Based on the response data received from the server, the device updates the display of the AI ​​character and provides a response to the user along with audio.

[0892] Step 8:

[0893] After the training is completed, the server analyzes the user's interaction history and generates detailed feedback and suggestions for improvement.

[0894] Input: Dialogue history

[0895] Output: Detailed feedback

[0896] How it works: The server analyzes data from completed training sessions and uses a generative AI model to generate feedback, including analysis of the user's speech and facial expressions.

[0897] Step 9:

[0898] The server generates feedback and sends it to the terminal for presentation to the user.

[0899] Input: Detailed feedback

[0900] Output: Feedback presented to the user

[0901] Specific operation: The device acquires feedback data and provides it to the user in text or voice.

[0902] Step 10:

[0903] The server stores the interaction history and feedback in a database for use in future training sessions.

[0904] Input: Dialogue history, feedback

[0905] Output: Saved data

[0906] Specific operation: The server stores the dialogue history and feedback in a database, and refers to this the next time a scenario is selected to customize it.

[0907] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0908] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0909] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0910] [Third embodiment]

[0911] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0912] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0913] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0914] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0915] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0916] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0917] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0918] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0919] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0920] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0921] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0922] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0923] The dialogue simulator system of the present invention begins when a user accesses the dialogue simulator system and selects a scenario. Once the user selects a scenario, the terminal transmits the selected scenario information to the server. The server reads data corresponding to the selected scenario from the database and customizes the scenario based on the user's past practice history.

[0924] The server then generates an AI character, which is configured to interact with the user, and the configuration data includes the character's appearance, personality, intonation, and facial expression patterns. The generated configuration data is sent to the device, which then displays it to the user.

[0925] When a user initiates a conversation, the device records the user's speech in real time and sends it to the server. The server analyzes the speech data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends this response data to the device. The device then provides a response to the user through an AI character, whose facial expressions and intonation also change according to the situation.

[0926] The server then analyzes the user's speech, facial expressions, and intonation in real time and generates feedback based on these. The feedback is then sent to the device and provided to the user in voice or text format. This feedback includes specific advice and suggestions for improvement.

[0927] Finally, the server stores the user's interaction history and feedback in a database, which is used for future practice and to analyze the user's progress and trends to customize the next scenario.

[0928] Specific examples

[0929] For example, if a user selects a scenario for improving presentation skills, the device sends "Improve presentation skills" to the server, and the server loads and customizes the scenario based on the associated template and past practice data.

[0930] To start a presentation, the user says, "Today I'm going to present about our new product." The device records this speech and sends it to the server. The server analyzes this information, generates a response such as, "Tell me more about your new product," and sends it back to the user via the device.

[0931] While the user continues the presentation, the server analyzes the user's speech content and intonation and generates appropriate feedback in real time. For example, after the presentation, specific feedback such as "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more" is provided.

[0932] Through these steps, users can effectively improve their presentation skills. This system provides an environment that is as close as possible to real conversations, and supports users in improving their skills.

[0933] The processing flow will be explained below.

[0934] Step 1:

[0935] The user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface.

[0936] Step 2:

[0937] The user selects the scenario of their choice (sales role-play, meeting practice, presentation skills improvement, etc.).

[0938] Step 3:

[0939] The terminal transmits the user's scenario selection information to the server.

[0940] Step 4:

[0941] The server reads data corresponding to the selected scenario from the database.

[0942] Step 5:

[0943] The server checks the user's past practice history and customizes the scenario.

[0944] Step 6:

[0945] The server generates configuration data for the AI ​​character, including the character's appearance, personality, intonation, and facial expression patterns.

[0946] Step 7:

[0947] The server sends the AI ​​character configuration data generated by the server to the device.

[0948] Step 8:

[0949] The device displays the AI ​​character to the user, preparing the user to begin interaction.

[0950] Step 9:

[0951] The user initiates the interaction and speaks the first word (e.g., beginning a presentation).

[0952] Step 10:

[0953] The device records the user's voice input and transmits the data to the server in real time.

[0954] Step 11:

[0955] The server analyzes the voice data and uses natural language processing to understand the user's intent.

[0956] Step 12:

[0957] The server generates an appropriate response based on the user's intentions, taking into account the character's personality and the scenario content.

[0958] Step 13:

[0959] The server generates a response and sends it to the terminal.

[0960] Step 14:

[0961] The device responds to the user in voice or text, changing the AI ​​character's facial expression and intonation.

[0962] Step 15:

[0963] The user then makes the next utterance and the dialogue continues. The device again records the voice input and sends it to the server.

[0964] Step 16:

[0965] The server analyzes the user's speech in real time and generates feedback based on the speech content, facial expressions, and speaking intonation.

[0966] Step 17:

[0967] The server generates feedback and sends it to the device.

[0968] Step 18:

[0969] The terminal provides the generated feedback to the user in real time by voice or text.

[0970] Step 19:

[0971] The server stores the user's interaction history and feedback in a database.

[0972] Step 20:

[0973] The server analyzes the user's learning status based on the stored data and generates data to customize the content of the next practice session.

[0974] Step 21:

[0975] The device will then display a report of the results to the user, including a score, specific areas for improvement, and suggestions for the next practice session.

[0976] Through the above steps, users can effectively improve their skills through interaction.

[0977] Example 1

[0978] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0979] Current dialogue simulation systems suffer from monotonous reactions to user utterances and a lack of appropriate feedback that takes into account the user's past practice history. This makes it difficult for users to achieve sufficient learning results, particularly in finer details such as pronunciation, intonation, and facial expressions, which can slow down improvement. Furthermore, the lack of advanced conversational simulation using generative AI models limits the user experience.

[0980] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0981] In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for a terminal to transmit the selected scenario information to the server; means for the server to read scenario data and the user's past practice history from a database and customize the scenario; means for the server to generate an AI character and transmit character setting data to the terminal; means for the terminal to record utterance data for the user to start a dialogue in real time and transmit it to the server; means for the server to analyze the voice data, understand the user's intention using natural language processing and generate an appropriate response; means for the server to transmit the generated response data to the terminal and provide the response to the user through the AI ​​character; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation, generate feedback, and transmit the generated feedback to the terminal and provide it to the user; and means for the server to store the dialogue history and feedback in a database and use it for future learning. This allows users to practice in an environment that more closely resembles actual dialogue and receive detailed feedback to improve their skills.

[0982] A "user" is a subject who uses the dialogue simulator system to select a scenario and engage in dialogue.

[0983] The "dialogue simulator system" is a system that simulates dialogue based on a scenario selected by the user and provides reactions and feedback from an AI character.

[0984] "Scenario" refers to a practice or scene setting selected by the user in the dialogue simulator system.

[0985] A "terminal" is a hardware device (for example, a personal computer or smartphone) that allows a user to access and operate the dialogue simulator system.

[0986] The "server" is the central component of the dialogue simulator system, and is a computer that loads and customizes scenario data, generates AI characters, analyzes user speech data, and generates responses.

[0987] "Scenario data" refers to data such as templates, storylines, and example questions and answers related to each scenario in the dialogue simulator system.

[0988] The "database" is a data storage system for storing scenario data, the user's past practice history, dialogue history, feedback, and so on.

[0989] "Customization" is the process of adaptively modifying and adjusting a scenario, taking into account the user's past practice history and other relevant data.

[0990] An "AI character" is an artificial intelligence-based character (avatar) generated in a dialogue simulator system to engage in dialogue with a user.

[0991] "Character setting data" refers to data that includes information such as the AI ​​character's appearance, personality, intonation, and facial expression patterns.

[0992] "Utterance data" refers to recorded data of voice utterances made by a user during a conversation.

[0993] "Natural language processing" is a technology that allows an AI character to understand what the user is saying and generate an appropriate response based on that.

[0994] "Response data" is data that includes a response or reaction generated by the server in response to the content of the user's utterance.

[0995] "Feedback" is evaluation information that includes advice and suggestions for improving the dialogue, and is generated based on the user's speech content, facial expressions, intonation, etc.

[0996] "Dialogue history" refers to a record of the dialogues that a user has had with the dialogue simulator system.

[0997] The dialogue simulator system of the present invention begins when a user accesses the dialogue simulator system and selects a scenario. The user accesses the system from a browser on a PC or smartphone, logs in, and then selects a scenario that suits their purpose from a list of scenarios provided. For example, the user can select the scenario "Improve Presentation Skills."

[0998] Once a selection is made, the device sends this information to the server via an HTTP request (for example, the POST method). At this time, the data sent includes the user ID and the selected scenario ID. On the server side, the corresponding scenario data is read from a database (for example, MySQL or PostgreSQL) based on the received scenario ID. At the same time, the user's past practice history is obtained and the scenario is customized based on this. This customization reflects the themes the user has previously experienced and the points that should be emphasized.

[0999] Next, the server generates an AI character. This AI character is generated in real time using, for example, a custom Unity API. The generated character's appearance, personality, intonation, facial expression patterns, etc. are configured. This configuration data is sent to the device in JSON format, and the device uses WebGL or WebRTC technology to display it on the screen.

[1000] When a conversation begins, the device records the user's speech in real time and performs noise cancellation and echo removal to obtain high-quality audio data. This audio data is sent to a server, where it is converted into text using speech recognition technology such as the Google Cloud Speech-to-Text API. The server then analyzes the user's intent using natural language processing with a generative AI model (e.g., BERT or GPT-3). Based on the results of this analysis, the server generates an appropriate response and sends the response data back to the device.

[1001] Based on the received response data, the device uses voice synthesis technology such as the Web Speech API to have the AI ​​character respond, changing facial expressions and intonation along with the response to achieve a more natural dialogue.

[1002] Furthermore, the server analyzes the user's speech content, facial expressions, and speaking intonation in real time, and provides specific feedback based on the results. This feedback may include detailed advice such as, "The product explanation section was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more." This feedback is sent from the server to the device, which then provides it to the user. This process is carried out using emotion analysis tools such as Microsoft Azure's Emotion API.

[1003] Finally, the server stores all interaction history and feedback in a database, which can be used for future practice sessions, to analyze the user's progress, and to provide more customized scenarios.

[1004] Examples of specific examples and prompts

[1005] For example, if a user selects the "Improve Presentation Skills" scenario, the scenario will include the following prompt:

[1006] "Please tell us three features of your new product."

[1007] "Please suggest appropriate responses to the audience questions."

[1008] "Please tell me how to effectively summarize at the end of a presentation."

[1009] When a user says, "Today I'm going to give a presentation about our new product," the device's microphone records this speech and sends it to the server, which then responds with, "Please tell us more about your new product." After the presentation, the server analyzes the user's speech and provides feedback such as, "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more."

[1010] This system allows users to improve even the finer details of pronunciation, intonation, and facial expressions through dialogue.

[1011] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1012] Step 1:

[1013] This is the procedure in which a user accesses the dialogue simulator system and selects a scenario.

[1014] Input: User login information, system access, scenario selection request.

[1015] Data processing and calculation: User authentication, scenario list display, confirmation of selected scenario.

[1016] Output: Selected scenario ID, user ID.

[1017] Specific operation: A user logs in from a PC or smartphone browser, browses the list of scenarios, and then selects a scenario such as "Improve Presentation Skills."

[1018] Step 2:

[1019] This is a procedure in which the terminal transmits scenario selection information to the server.

[1020] Input: Selected scenario ID, user ID.

[1021] Data processing and calculation: Generating and sending HTTP requests (POST method, etc.).

[1022] Output: The scenario selection request sent to the server.

[1023] Specific operation: The device generates an HTTP request including the selected scenario ID and user ID and sends it to the server.

[1024] Step 3:

[1025] This is the procedure in which the server reads scenario data and the user's past practice history from the database and customizes the scenario.

[1026] Input: Scenario ID, User ID.

[1027] Data processing and calculation: Performing database queries, filtering and extracting past exercise history, and adaptive scenario customization.

[1028] Output: Customized scenario data.

[1029] Specific operation: The server reads the relevant scenario data and the user's past practice history from the MySQL database, and customizes the scenario based on the previous practice content.

[1030] Step 4:

[1031] This is the procedure in which the server generates an AI character and sends the character's setting data to the terminal.

[1032] Input: Customized scenario data.

[1033] Data processing and calculation: Calling the AI ​​character generation module and generating setting data.

[1034] Output: AI character configuration data (JSON format).

[1035] How it works: The server uses Unity's custom API to generate a character, create settings such as appearance, personality, intonation, and facial expression patterns, and send them to the device.

[1036] Step 5:

[1037] This is the procedure in which the device displays an AI character and begins a dialogue with the user.

[1038] Input: AI character configuration data.

[1039] Data processing and calculation: Reading setting data, using the screen drawing library.

[1040] Output: An AI character displayed on the screen.

[1041] Specific operation: The device displays an AI character on the screen based on the configuration data received using WebGL and WebRTC technology, and the conversation begins when the user presses the "Start" button.

[1042] Step 6:

[1043] This is a procedure in which the terminal records the user's speech data and transmits it to the server.

[1044] Input: User utterance.

[1045] Data processing and calculation: voice recording, noise cancellation, echo removal.

[1046] Output: Preprocessed audio data.

[1047] Specific operation: When a user speaks to an AI character, the device's microphone records the voice, performs noise cancellation and echo removal in real time, and then sends the voice data to the server.

[1048] Step 7:

[1049] This is the procedure where the server analyzes the voice data and generates an appropriate response.

[1050] Input: Preprocessed audio data.

[1051] Data processing and calculation: Speech-to-Text, natural language processing (BERT and GPT-3).

[1052] Output: Response data (text format).

[1053] How it works: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text, then uses a natural language processing model to analyze the user's intent and generate an appropriate response.

[1054] Step 8:

[1055] This is the procedure by which the device displays a response to the user through an AI character.

[1056] Input: Response data from the server (text format).

[1057] Data processing and calculation: voice synthesis, facial animation generation.

[1058] Output: Voice responses and facial expressions by the AI ​​character.

[1059] How it works: The device uses the Web Speech API to convert the response into voice and responds to the user along with the AI ​​character's facial expressions, resulting in a more natural dialogue.

[1060] Step 9:

[1061] This is a procedure in which the server analyzes the user's speech and generates real-time feedback.

[1062] Input: User utterance and parsed results.

[1063] Data processing and calculation: Sentiment analysis, speech evaluation, and extraction of areas for improvement.

[1064] Output: Feedback data.

[1065] Specific operation: The server uses Microsoft Azure's Emotion API and emotion analysis tools to evaluate the user's speech and generate feedback including specific improvements and advice.

[1066] Step 10:

[1067] This is a procedure in which the terminal provides feedback to the user.

[1068] Input: Feedback data.

[1069] Data processing and calculation: Display of feedback contents.

[1070] Output: The displayed feedback.

[1071] Specific operation: The device provides the received feedback data to the user by voice or text, using a text box displayed on the screen or a speaker for voice feedback.

[1072] Step 11:

[1073] This is the procedure by which the server stores the dialogue history and feedback in a database.

[1074] Input: Dialogue history, feedback data.

[1075] Data processing and calculation: Writing to database, saving data.

[1076] Output: Stored interaction history and feedback.

[1077] What it does: The server stores all interaction history and feedback in a MySQL database, which is used to customize the scenario for the next session.

[1078] (Application example 1)

[1079] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1080] Previously, training to improve customer service skills in brick-and-mortar stores required effort and cost to deal with real customers, and had the problem of not being able to receive immediate feedback. Furthermore, there was a lack of opportunities for new staff to practice interacting with real customers, making it difficult for them to improve their skills. For this reason, there is a need for a system that can improve customer service skills while receiving effective, real-time feedback.

[1081] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1082] In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and send character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and send it to the server; means for the server to analyze the user's utterance data to generate an appropriate response and send it to the terminal; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation and generate feedback; means for the server to send the generated feedback to the terminal and provide it to the user; means for the server to save the user's dialogue history and feedback and use it for future learning; means for the terminal to function as a customer service simulator and enable brick-and-mortar store staff to improve their customer service skills; and means for providing interactive feedback to the user. This enables brick-and-mortar store staff to improve their customer service skills by receiving feedback effectively and in real time.

[1083] A "dialogue simulator system" is a system that allows users to improve specific skills or knowledge by engaging in dialogue in a simulated format.

[1084] A "scenario" is a set of stories or conditions designed to simulate a particular situation or dialogue.

[1085] A "server" is a computer system that serves as the core of the dialogue simulator system accessed by users, and is a device that processes data and generates AI characters.

[1086] "Means for reading data and customizing a scenario" refers to a mechanism in which the server reads related information from a database for a scenario selected by a user and modifies it to suit the user's needs.

[1087] An "AI character" is a virtual character with artificial intelligence that is generated in a dialogue simulator to have an interactive conversation with a user.

[1088] "Configuration data" is data that defines the specific attributes and behavior of an AI character, such as their appearance, personality, intonation, and facial expression patterns.

[1089] "Utterance data" is data of the voice uttered by the user during a conversation.

[1090] A "reply" is a message that the server generates in response to an analysis of the user's speech data.

[1091] "Expressions" refer to the facial movements and emotional expressions shown by AI characters in the dialogue simulator.

[1092] "Intonation" refers to the rise and fall of the voice when speaking.

[1093] "Feedback" refers to information such as evaluations and areas for improvement provided based on the user's interactions and actions.

[1094] "Dialogue history" is a record of past dialogues between the user and the AI ​​character.

[1095] The "customer service simulator" is a system that allows store staff to simulate interactions with customers and improve their response skills.

[1096] "Interactive feedback" refers to dynamic evaluations and advice provided in real time in response to a user's actions.

[1097] This invention relates to a customer interaction simulator system that allows store staff to improve their customer interaction skills. The system starts when a user accesses the interaction simulator system and selects a scenario. Based on the selected scenario, the server loads data and customizes the scenario. The server then generates an AI character and sends its setting data to the terminal.

[1098] After the user initiates a dialogue, the device receives the user's speech data and sends it to the server. The server analyzes the user's speech data, generates an appropriate response using a generative AI model, and sends this response data to the device. The device then provides a response to the user through an AI character. At this time, the AI ​​character's facial expression and intonation also change according to the scenario. The server then analyzes the content of the user's speech, facial expression, and speaking intonation to generate feedback. The generated feedback is sent to the device and provided to the user.

[1099] Hardware and software used

[1100] Hardware:

[1101] Smart glasses (e.g., smart devices)

[1102] Head-mounted displays (e.g., mixed reality devices)

[1103] software:

[1104] Python (programming language)

[1105] speech_recognition (speech recognition library)

[1106] transformers (natural language processing library)

[1107] The server captures the user's voice input data and converts it into text using a speech recognition library. The converted data is then used by an AI character to generate an appropriate response using a natural language processing library. This response is then provided to the user via the device. The user's conversation is analyzed in real time and feedback is generated. This feedback includes specific advice and suggestions for improvement and is provided to the user interactively via the device.

[1108] Specific examples

[1109] For example, suppose a staff member at a brick-and-mortar store selects the "Basic Customer Response Scenario" and starts training. The staff member says, "Hello, is there anything you'd like to buy in our store?" The device records this speech, and the server analyzes the audio data. Based on the analysis results, the server generates a response such as "Hello, is there anything in particular you'd like to buy today?" and displays the response through the device.

[1110] The system then analyzes the speech and intonation of the staff member and provides feedback such as, "If you ask your question with a more emphasized intonation, it will be easier for the customer to understand." This feedback is provided interactively in real time, helping the staff member improve their skills.

[1111] In this way, store staff can effectively improve their customer service skills.

[1112] Example prompt sentence:

[1113] "Hello, are you looking for something in the store?"

[1114] "Is there anything in particular you're looking for today?"

[1115] "If you ask questions with a bit more emphasis on intonation, you'll be able to communicate more clearly to the customer."

[1116] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1117] Step 1:

[1118] A user accesses the dialogue simulator system and selects a scenario. As input, the user accesses the scenario selection interface from the device they use (smart glasses or a head-mounted display) and selects a specific scenario (e.g., "Basic Customer Interaction Scenario"). As output, the selected scenario information is sent to the server.

[1119] Step 2:

[1120] The server reads the data corresponding to the selected scenario and customizes the scenario. The input is the scenario selection information sent by the user, and based on this the server retrieves related information from the database. The data is processed by customizing the scenario based on the user's past practice history, and customized scenario data is generated as output.

[1121] Step 3:

[1122] The server generates an AI character and sends the character's configuration data to the device. The input is customized scenario data, and the AI ​​character's configuration (appearance, personality, intonation, facial expression patterns, etc.) is generated. The data is processed using a generative AI model to generate the character, and the configuration data is generated as output and sent to the device.

[1123] Step 4:

[1124] The user starts a conversation. The device receives the user's speech data and sends it to the server. The input is the user's voice data at the start of the conversation, which the device captures in real time. The data is then processed using a speech recognition library to convert the voice data into text data, and the converted text data is generated as output and sent to the server.

[1125] Step 5:

[1126] The server analyzes the user's speech data, generates an appropriate response, and sends it to the device. The converted text data is used as input, and a response is generated using a generative AI model based on this. Natural language processing is used to process the data, analyzing the user's intent, and an appropriate response data is generated as output and sent to the device.

[1127] Step 6:

[1128] The device provides the user with a response from an AI character. The response data sent from the server is input, and the device receives this. The data is processed by changing the AI ​​character's facial expression and intonation based on the settings, and the response is provided to the user visually and audibly as output.

[1129] Step 7:

[1130] The server analyzes the user's speech content, facial expressions, and speaking intonation to generate feedback. The input is the user's latest speech data and dialogue history, and data calculations are performed using an AI model based on this data. The output is feedback that includes specific advice and suggestions for improvement.

[1131] Step 8:

[1132] The server sends the generated feedback to the terminal and provides it to the user. The generated feedback data is input and sent to the terminal. The data is processed by adjusting the display format of the feedback, and the feedback is provided to the user in an interactive form as output.

[1133] Step 9:

[1134] The server stores the user's interaction history and feedback and uses it for future learning. The input is the latest interaction history and feedback data, and the data is processed and stored appropriately in a database. The output is used to customize future scenarios and improve the learning algorithm.

[1135] Example prompt sentence:

[1136] "Hello, are you looking for something in the store?"

[1137] "Is there anything in particular you're looking for today?"

[1138] "If you ask questions with a bit more emphasis on intonation, you'll be able to communicate more clearly to the customer."

[1139] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1140] The dialogue simulator system of the present invention is structured so that a user accesses the system and selects a scenario, and the entire system is processed mainly by the server. The specific flow of processing will be explained below.

[1141] When starting to use the system, the user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface. The user selects the desired scenario (e.g., sales role-play, meeting practice, presentation skill improvement, etc.), and the selection information is sent from the terminal to the server.

[1142] The server reads data corresponding to the selected scenario from the database and customizes the scenario based on the user's past practice history. Based on this customized scenario, the server generates configuration data for the AI ​​character and sends the configuration data to the device. The device displays the generated AI character to the user, ready for the user to begin interacting with it.

[1143] When a user initiates a dialogue and speaks, the device records the user's voice input and sends it to the server in real time. The server analyzes the voice data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends the response data to the device. The device responds to the user by changing the AI ​​character's facial expression and intonation.

[1144] Furthermore, the system uses an emotion engine to analyze the user's speech and facial expression data. The server uses the emotion engine to recognize the user's emotions and adjusts the response and feedback accordingly. For example, if the user is nervous, the AI ​​character will respond in an encouraging manner to calm the user.

[1145] The server also uses the emotion engine to generate feedback. It analyzes the user's speech content, facial expressions, and speaking intonation in real time, and generates specific feedback based on the analysis results. This feedback is sent to the device and provided to the user in the form of voice or text. The feedback includes specific advice and areas for improvement.

[1146] For example, suppose a user selects a scenario for improving presentation skills and begins a presentation. When the user says, "Today I'm going to give a presentation about our new product," the device records this speech and sends it to the server. The server analyzes this speech data and generates a response such as, "Tell me more about your new product." It then analyzes the user's emotions using an emotion engine and adjusts the tone and content of the response as needed.

[1147] After the presentation, the server analyzes the user's speech and intonation, and provides specific feedback such as, "The product explanation was particularly easy to understand in the presentation. However, it would be even better if you emphasized the intonation a little more." The feedback is also adjusted appropriately, taking into account the user's emotions.

[1148] Finally, the server stores the user's dialogue history and feedback in a database and uses it as data to personalize the next scenario. In this way, the dialogue simulator system of the present invention helps users improve their skills and provides a more effective training environment.

[1149] To implement the invention, a program is required to smoothly carry out a series of operations, including user selection, data analysis, emotion recognition, and feedback generation, allowing users to effectively improve their practical skills through realistic interactions.

[1150] The processing flow will be explained below.

[1151] Step 1:

[1152] The user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface.

[1153] Step 2:

[1154] The user selects the scenario of their choice (e.g., sales role-play, meeting practice, presentation skill improvement, etc.).

[1155] Step 3:

[1156] The terminal transmits the user's scenario selection information to the server.

[1157] Step 4:

[1158] The server reads data corresponding to the selected scenario from the database.

[1159] Step 5:

[1160] The server reviews the user's past practice history and customizes the scenario, which can include adjusting the practice level or focusing on specific tasks.

[1161] Step 6:

[1162] The server generates the AI ​​character's configuration data, which includes the character's appearance, personality, intonation, and facial expression patterns generated by the emotion engine.

[1163] Step 7:

[1164] The server sends the AI ​​character configuration data generated by the server to the device.

[1165] Step 8:

[1166] The device displays the AI ​​character to the user and prepares to begin a conversation.

[1167] Step 9:

[1168] The user initiates the interaction and speaks the first word (e.g., beginning a presentation).

[1169] Step 10:

[1170] The device records the user's voice input in real time and sends the data to the server.

[1171] Step 11:

[1172] The server receives the recorded data, performs voice analysis, and analyzes the user's linguistic intent.

[1173] Step 12:

[1174] The server uses an emotion engine to analyze the facial expression data collected at the same time as the user's speech data, and recognizes the user's emotions.

[1175] Step 13:

[1176] The server generates an appropriate response based on the user's intentions and emotions, and adjusts the character's facial expression and intonation.

[1177] Step 14:

[1178] The server sends the generated response data to the terminal.

[1179] Step 15:

[1180] The device changes the AI ​​character's facial expression and intonation to provide responses to the user in voice or text.

[1181] Step 16:

[1182] The user makes the next utterance and the dialogue proceeds. The device again records the user's voice input and sends it to the server.

[1183] Step 17:

[1184] The server analyzes the user's continuous speech data and facial expression data, and generates feedback in real time based on the user's emotions and speech content.

[1185] Step 18:

[1186] The server transmits the generated feedback data to the terminal.

[1187] Step 19:

[1188] The device provides real-time feedback to the user via voice or text, providing suggestions for improvement and advice.

[1189] Step 20:

[1190] The server stores the user's interaction history and feedback in a database.

[1191] Step 21:

[1192] The server analyzes the user's learning status based on the stored data and generates data to customize the content of the next practice session.

[1193] Step 22:

[1194] The device will then display a report of the results to the user, including a score for the exercise, specific areas for improvement, and advice for the next exercise.

[1195] Through these detailed steps, users can effectively improve their skills while experiencing realistic interactions. By utilizing the emotion engine, appropriate responses and feedback are provided that are tailored to the user's emotions, creating a more natural and effective learning environment.

[1196] Example 2

[1197] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1198] Conventional dialogue simulator systems lack real-time feedback on user voice input and dialogue content, making it difficult to customize specific scenarios that effectively reflect the user's emotions and practice history.Furthermore, it is difficult to provide a realistic dialogue environment by changing facial expressions and intonation during dialogue.

[1199] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and transmit character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and transmit it to the server; means for the server to analyze the user's utterance data, generate an appropriate response, and transmit it to the terminal; means for the terminal to respond while changing the AI ​​character's facial expression and intonation; means for the server to recognize the user's emotions using an emotion engine and adjust the response content and feedback based on the recognition; means for transmitting the feedback generated by the server to the terminal and providing it to the user; and means for the server to save the user's dialogue history and feedback and use it for future learning. This enables advanced scenario customization that reflects the user's practice history and emotions and feedback with real-time emotion recognition.

[1200] A "simulator system" is a computer-based system for conducting interactive training for a user.

[1201] A "user" is an individual who uses this system to select a scenario and conduct dialogue training.

[1202] A "server" is a computer that centrally processes the entire system, analyzes input data from users, and generates appropriate scenarios and responses.

[1203] A "terminal" is a device through which a user accesses the system and uses the functions of the dialogue simulator system.

[1204] A "scenario" refers to a specific situation or setting prepared for dialogue training.

[1205] A "database" is a system for managing and storing data used by the dialogue simulator system.

[1206] "Customization" refers to adjusting and changing scenarios and settings based on a user's past history and specific needs.

[1207] An "AI character" is a character with artificial intelligence that converses with the user within the dialogue simulator system.

[1208] "Configuration data" is information used to determine the appearance, tone of voice, and behavior of an AI character.

[1209] "Utterance data" is data of the voice uttered by the user during a conversation.

[1210] "Natural language processing" is a technology that enables computers to understand and generate human language.

[1211] An "emotion engine" is software that analyzes a user's voice and facial expressions to determine their emotional state.

[1212] "Feedback" refers to evaluations and advice given to a user's interactions and actions.

[1213] The "dialogue history" is a record of the content of dialogues and scenarios that the user has had in the past.

[1214] "Intonation" refers to the intonation and pitch of the voice when speaking.

[1215] "Facial expressions" are expressions on the user's face that indicate emotions and reactions.

[1216] The dialogue simulator system of the present invention is structured so that the user accesses the system and selects a scenario, and the entire system is processed by a server. The system analyzes the user's speech data, facial expression data, and dialogue history to provide an environment for advanced dialogue with the user.

[1217] The specific hardware required to implement this system includes the devices used by users (e.g., PCs, smartphones, tablets) and the server that controls the entire system. The main software used includes a speech recognition API (e.g., Google Cloud Speech-to-Text API), a natural language processing module, an emotion recognition API (e.g., Microsoft Azure Cognitive Services Emotion API), and a module for AI character generation.

[1218] When starting to use the system, the user accesses the dialogue simulator system through their own device and checks the scenario selection menu on the interface. When the user selects the desired scenario, the selection information is sent from the device to the server. The server reads data based on the selected scenario from the database and customizes the scenario by referring to the user's past practice history. Based on the customized scenario, the server generates configuration data for the AI ​​character and sends it to the device.

[1219] When a user initiates a dialogue, the device records the user's voice input in real time and sends it to the server. The server analyzes the voice data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends the response data to the device. The device responds to the user by changing the AI ​​character's facial expression and intonation.

[1220] In addition, the server uses an emotion engine to recognize the user's emotions and adjusts responses and feedback accordingly. For example, if the user is nervous, the AI ​​character will respond in an encouraging manner to ease the user's tension. The server also generates specific feedback based on the results of analyzing the user's speech and facial expression data and sends it to the device. The generated feedback is provided to the user in the form of voice or text, and includes specific advice and areas for improvement.

[1221] Furthermore, the server stores the user's dialogue history and feedback in a database. This stored data is used as personalized data for the next scenario customization, allowing the user to continuously receive dialogue training that is individually adapted to their needs.

[1222] As a concrete example, let's say a user selects the "Improve Presentation Skills" scenario and begins a presentation. When the user says, "Today I'm going to give a presentation about our new product," the device records this speech and sends it to the server. The server analyzes the audio data and generates a response such as, "Tell me more about your new product." At this time, it analyzes the user's emotions using an emotion engine and adjusts the tone and content of the response as necessary. After the presentation ends, the server analyzes the content and intonation of the user's speech and provides specific feedback such as, "The part of your presentation that was particularly easy to understand was the product explanation. However, it would be even better if you emphasized your intonation a little more."

[1223] An example of a prompt in this system is, "You have selected a scenario for improving your presentation skills and have begun your presentation. Please say, 'Today I will be presenting about our new product.'"

[1224] This allows the dialogue simulator system to help users improve their skills and provide a realistic practice environment.

[1225] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1226] Step 1:

[1227] The user accesses the dialogue simulator system and selects a scenario.

[1228] Specific operation: The user opens the browser on the device, accesses the system URL, and selects "Improve Presentation Skills" from the scenario menu.

[1229] Input: Scenario information selected by the user when they access the system.

[1230] Output: The selected scenario information is sent from the terminal to the server.

[1231] Step 2:

[1232] The server loads data corresponding to the selected scenario and customizes the scenario.

[1233] Specific operation: The server loads the "Improve Presentation Skills" scenario template from the database. It then refers to the user's past history and adjusts and modifies the scenario.

[1234] Input: Selected scenario information and user practice history data.

[1235] Output: Customized scenario data.

[1236] Step 3:

[1237] The server generates configuration data for the AI ​​character and sends it to the device.

[1238] Specific operation: Set the AI ​​character's name, appearance, tone of voice, etc., and send the setting data to the device in JSON format.

[1239] Input: Customized scenario data.

[1240] Output: AI character configuration data is sent to the device.

[1241] Step 4:

[1242] The user starts a dialogue, and the terminal receives the utterance data and transmits it to the server.

[1243] Specific operation: The user says, "Today I will be giving a presentation about our new product." The device records the voice and sends it to the server.

[1244] Input: User's voice input.

[1245] Output: The recorded audio data is sent to the server.

[1246] Step 5:

[1247] The server analyzes the user's voice data, generates an appropriate response, and sends it to the terminal.

[1248] Specific operation: The voice data is converted to text using the Google Cloud Speech-to-Text API, and the text data is analyzed using a natural language processing module to generate a response.

[1249] Input: Recorded audio data.

[1250] Output: Response data such as "Please tell me more about your new product" is generated and sent to the terminal.

[1251] Step 6:

[1252] The device responds by changing the AI ​​character's facial expression and intonation.

[1253] Specific operation: The text response is converted into speech using a speech synthesis API, and the response is played back as speech while the AI ​​character's facial expression changes.

[1254] Input: Response data.

[1255] Output: A voice-synthesized response and facial expression changes are provided to the user.

[1256] Step 7:

[1257] The server uses an emotion engine to recognize the user's emotions and adjusts responses and feedback accordingly.

[1258] Specific behavior: Analyzes emotions using the emotion engine (Microsoft Azure Cognitive Services Emotion API) and adjusts the tone and content of responses.

[1259] Input: User's voice and facial expression data.

[1260] Output: The adjusted response data.

[1261] Step 8:

[1262] The server generates feedback and sends it to the terminal for presentation to the user.

[1263] Specific actions: Analyze the user's speech content and intonation and generate specific feedback, such as, "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more."

[1264] Input: Speech content, intonation, and facial expression data.

[1265] Output: The generated feedback data is sent to the terminal and provided to the user.

[1266] Step 9:

[1267] The server stores the user's interaction history and feedback and uses it for future learning.

[1268] Specific operation: After the dialogue session ends, the spoken content and emotional data are saved in a database. The next time a scenario is customized, an individually adapted scenario is generated based on the saved data.

[1269] Input: User interaction history and feedback data.

[1270] Output: Stored interaction history and feedback data.

[1271] (Application example 2)

[1272] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1273] In today's brick-and-mortar stores, improving the customer service skills of store associates is important, but there is a lack of systems for effectively training them. Furthermore, traditional training methods lack consistency and objectivity, making it difficult to provide individualized instruction tailored to each employee's strengths and weaknesses.

[1274] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and send character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and send it to the server; means for the server to analyze the user's utterance data, generate an appropriate response, and send it to the terminal; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation and generate feedback; means for the server to send the generated feedback to the terminal and provide it to the user; means for the server to save the user's dialogue history and feedback and use it for future learning; means for providing customer service training scenarios to the terminal, including a function to perform emotion analysis in real time and adjust responses; and means for the terminal to present detailed feedback and areas for improvement after the training is completed. This enables consistent, high-quality customer service training and effectively supports employee skill improvement.

[1275] A "dialogue simulator system" is a system in which the user selects a scenario and engages in dialogue with an AI character based on that scenario.

[1276] A "scenario" is a set situation or theme for a user to practice with the dialogue simulator.

[1277] The "server" is a central processing unit that processes the scenarios and speech data selected by the user, generates AI characters, and provides feedback.

[1278] An "AI character" is a virtual character equipped with artificial intelligence that is generated in a dialogue simulator system to engage in dialogue with a user.

[1279] A "terminal" is a computer device that allows a user to access the dialogue simulator system and select a scenario or engage in dialogue.

[1280] "Utterance data" is voice data of what the user says in the dialogue simulator system.

[1281] "Feedback" is information indicating advice and areas for improvement provided based on the analysis of the user's dialogue content, facial expressions, and speaking style.

[1282] "Emotion analysis" is a technology that analyzes a user's emotional state from their speech and facial expressions.

[1283] A "customer service training scenario" is a specific interaction situation designed to help store staff improve their customer service skills.

[1284] "History" is a record of the interactions a user has had and feedback they have received in the past.

[1285] "Consistency" means that customer service training using the dialogue simulator system is always conducted with the same quality.

[1286] "High-quality customer service training" refers to providing effective training to improve customer service skills that is tailored to the strengths and weaknesses of each individual store employee.

[1287] The embodiment of the present invention is aimed at "customer service training" and specifically includes a step in which a user utilizes a dialogue simulator system to improve customer service skills.

[1288] First, a user accesses the dialogue simulator system using a device (e.g., a smartphone) and selects a specific scenario for training (e.g., introducing a new product or handling a complaint). The device then sends the selection information to the server, which then reads the scenario data from a database and customizes the scenario based on the user's past practice history.

[1289] Next, the server generates an AI character based on the customized scenario and sends the character's configuration data to the device. The device then prepares for the user to begin a dialogue, and the user speaks to simulate a real dialogue. For example, a user who selected a new product introduction scenario might say, "Tell me about the new products that have just arrived."

[1290] As the conversation progresses, the device sends the user's speech data in real time to the server, which then analyzes it using a natural language processing (NLP) engine. Specifically, it uses the Google Cloud Natural Language API to understand what the user is saying and generate appropriate responses based on that. This process also utilizes generative AI models.

[1291] In addition to generating responses, the server uses Affectiva's emotion recognition engine to analyze the user's emotional state and adjust the AI ​​character's facial expressions and intonation accordingly. The server also generates detailed feedback based on the analysis of the speech and facial expression data, suggesting areas for improvement to the user. This feedback includes specific advice and areas for improving effective customer service skills. After training is complete, the feedback is returned to the terminal and presented to the user.

[1292] The server stores this dialogue history and feedback in a database and uses it as data to make the next training session more effective.The next time a scenario is selected, it is automatically customized based on past history, providing the user with the optimal training environment.

[1293] For example, if a user says, "Today I'm going to give a presentation about our new product," the server will respond with, "Tell me more about your new product." If the user is nervous, the AI ​​character will offer words of encouragement, such as, "Relax and let's talk. Let's do our best together."

[1294] Examples of prompts:

[1295] "New product introduction"

[1296] User: "Tell me about the new items you've got in stock."

[1297] AI Character: "This new product uses the latest technology and offers 50% longer battery life than the previous model. Plus, it has a new, stylish design."

[1298] In this way, the dialogue simulator system effectively supports the improvement of customer service skills in brick-and-mortar stores, enabling store clerks to provide high-quality service in actual customer service situations.

[1299] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1300] Step 1:

[1301] A user accesses a terminal and starts the dialogue simulator system. A scenario selection menu is displayed so that the user can select a specific customer service training scenario.

[1302] Input: User's scenario selection

[1303] Output: Selected scenario information

[1304] Specific operation: The terminal acquires the scenario information selected by the user from the menu, formats the selected information, and sends it to the server.

[1305] Step 2:

[1306] Based on the scenario selection information received by the server, the server reads the corresponding scenario data from the database and customizes the scenario based on the user's past practice history.

[1307] Input: Selected scenario information, user's past practice history

[1308] Output: Customized scenario data

[1309] Specific operation: The server queries the database to obtain the user's practice history and selected scenario data. Based on this data, it customizes the scenario using scripts and a rules engine.

[1310] Step 3:

[1311] The server generates an AI character based on the customized scenario data and sends the character's setting data to the device.

[1312] Input: Customized scenario data

[1313] Output: AI character configuration data

[1314] Specific operation: The generative AI model is used to generate dialogue and facial expression data for the AI ​​character, and the setting data is sent to the device.

[1315] Step 4:

[1316] The user starts a conversation with the AI ​​character on their device. The user's speech is recorded by the device's microphone and sent to the server in real time.

[1317] Input: User utterance

[1318] Output: Sending speech data to the server

[1319] Specific operation: The device records the user's speech as audio data, converts it into an appropriate format, and sends it to the server.

[1320] Step 5:

[1321] The server analyzes the user's speech data using a natural language processing engine (Google Cloud Natural Language API) and generates an appropriate response.

[1322] Input: User utterance data

[1323] Output: The generated response

[1324] How it works: The server inputs the speech data into a natural language processing engine, analyzes the user's intent, and uses a generative AI model to generate an appropriate response.

[1325] Step 6:

[1326] The server analyzes the user's speech and facial expression data using an emotion analysis engine (Affectiva) and adjusts the AI ​​character's facial expressions and intonation.

[1327] Input: Speech data, facial expression data

[1328] Output: Adjusted response data

[1329] How it works: The server inputs speech data and facial expression data into an emotion analysis engine to identify the user's emotional state, and then adjusts the AI ​​character's facial expression and intonation accordingly.

[1330] Step 7:

[1331] The server sends the response and analysis results to the terminal, which displays them to the user.

[1332] Input: Adjusted response data

[1333] Output: The response that is displayed to the user

[1334] Specific operation: Based on the response data received from the server, the device updates the display of the AI ​​character and provides a response to the user along with audio.

[1335] Step 8:

[1336] After the training is completed, the server analyzes the user's interaction history and generates detailed feedback and suggestions for improvement.

[1337] Input: Dialogue history

[1338] Output: Detailed feedback

[1339] How it works: The server analyzes data from completed training sessions and uses a generative AI model to generate feedback, including analysis of the user's speech and facial expressions.

[1340] Step 9:

[1341] The server generates feedback and sends it to the terminal for presentation to the user.

[1342] Input: Detailed feedback

[1343] Output: Feedback presented to the user

[1344] Specific operation: The device acquires feedback data and provides it to the user in text or voice.

[1345] Step 10:

[1346] The server stores the interaction history and feedback in a database for use in future training sessions.

[1347] Input: Dialogue history, feedback

[1348] Output: Saved data

[1349] Specific operation: The server stores the dialogue history and feedback in a database, and refers to this the next time a scenario is selected to customize it.

[1350] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1351] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1352] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1353] [Fourth embodiment]

[1354] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1355] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1356] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1357] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1358] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1359] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1360] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1361] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1362] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1363] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1364] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1365] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1366] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1367] The dialogue simulator system of the present invention begins when a user accesses the dialogue simulator system and selects a scenario. Once the user selects a scenario, the terminal transmits the selected scenario information to the server. The server reads data corresponding to the selected scenario from the database and customizes the scenario based on the user's past practice history.

[1368] The server then generates an AI character, which is configured to interact with the user, and the configuration data includes the character's appearance, personality, intonation, and facial expression patterns. The generated configuration data is sent to the device, which then displays it to the user.

[1369] When a user initiates a conversation, the device records the user's speech in real time and sends it to the server. The server analyzes the speech data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends this response data to the device. The device then provides a response to the user through an AI character, whose facial expressions and intonation also change according to the situation.

[1370] The server then analyzes the user's speech, facial expressions, and intonation in real time and generates feedback based on these. The feedback is then sent to the device and provided to the user in voice or text format. This feedback includes specific advice and suggestions for improvement.

[1371] Finally, the server stores the user's interaction history and feedback in a database, which is used for future practice and to analyze the user's progress and trends to customize the next scenario.

[1372] Specific examples

[1373] For example, if a user selects a scenario for improving presentation skills, the device sends "Improve presentation skills" to the server, and the server loads and customizes the scenario based on the associated template and past practice data.

[1374] To start a presentation, the user says, "Today I'm going to present about our new product." The device records this speech and sends it to the server. The server analyzes this information, generates a response such as, "Tell me more about your new product," and sends it back to the user via the device.

[1375] While the user continues the presentation, the server analyzes the user's speech content and intonation and generates appropriate feedback in real time. For example, after the presentation, specific feedback such as "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more" is provided.

[1376] Through these steps, users can effectively improve their presentation skills. This system provides an environment that is as close as possible to real conversations, and supports users in improving their skills.

[1377] The processing flow will be explained below.

[1378] Step 1:

[1379] The user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface.

[1380] Step 2:

[1381] The user selects the scenario of their choice (sales role-play, meeting practice, presentation skills improvement, etc.).

[1382] Step 3:

[1383] The terminal transmits the user's scenario selection information to the server.

[1384] Step 4:

[1385] The server reads data corresponding to the selected scenario from the database.

[1386] Step 5:

[1387] The server checks the user's past practice history and customizes the scenario.

[1388] Step 6:

[1389] The server generates configuration data for the AI ​​character, including the character's appearance, personality, intonation, and facial expression patterns.

[1390] Step 7:

[1391] The server sends the AI ​​character configuration data generated by the server to the device.

[1392] Step 8:

[1393] The device displays the AI ​​character to the user, preparing the user to begin interaction.

[1394] Step 9:

[1395] The user initiates the interaction and speaks the first word (e.g., beginning a presentation).

[1396] Step 10:

[1397] The device records the user's voice input and transmits the data to the server in real time.

[1398] Step 11:

[1399] The server analyzes the voice data and uses natural language processing to understand the user's intent.

[1400] Step 12:

[1401] The server generates an appropriate response based on the user's intentions, taking into account the character's personality and the scenario content.

[1402] Step 13:

[1403] The server generates a response and sends it to the terminal.

[1404] Step 14:

[1405] The device responds to the user in voice or text, changing the AI ​​character's facial expression and intonation.

[1406] Step 15:

[1407] The user then makes the next utterance and the dialogue continues. The device again records the voice input and sends it to the server.

[1408] Step 16:

[1409] The server analyzes the user's speech in real time and generates feedback based on the speech content, facial expressions, and speaking intonation.

[1410] Step 17:

[1411] The server generates feedback and sends it to the device.

[1412] Step 18:

[1413] The terminal provides the generated feedback to the user in real time by voice or text.

[1414] Step 19:

[1415] The server stores the user's interaction history and feedback in a database.

[1416] Step 20:

[1417] The server analyzes the user's learning status based on the stored data and generates data to customize the content of the next practice session.

[1418] Step 21:

[1419] The device will then display a report of the results to the user, including a score, specific areas for improvement, and suggestions for the next practice session.

[1420] Through the above steps, users can effectively improve their skills through interaction.

[1421] Example 1

[1422] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1423] Current dialogue simulation systems suffer from monotonous reactions to user utterances and a lack of appropriate feedback that takes into account the user's past practice history. This makes it difficult for users to achieve sufficient learning results, particularly in finer details such as pronunciation, intonation, and facial expressions, which can slow down improvement. Furthermore, the lack of advanced conversational simulation using generative AI models limits the user experience.

[1424] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1425] In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for a terminal to transmit the selected scenario information to the server; means for the server to read scenario data and the user's past practice history from a database and customize the scenario; means for the server to generate an AI character and transmit character setting data to the terminal; means for the terminal to record utterance data for the user to start a dialogue in real time and transmit it to the server; means for the server to analyze the voice data, understand the user's intention using natural language processing and generate an appropriate response; means for the server to transmit the generated response data to the terminal and provide the response to the user through the AI ​​character; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation, generate feedback, and transmit the generated feedback to the terminal and provide it to the user; and means for the server to store the dialogue history and feedback in a database and use it for future learning. This allows users to practice in an environment that more closely resembles actual dialogue and receive detailed feedback to improve their skills.

[1426] A "user" is a subject who uses the dialogue simulator system to select a scenario and engage in dialogue.

[1427] The "dialogue simulator system" is a system that simulates dialogue based on a scenario selected by the user and provides reactions and feedback from an AI character.

[1428] "Scenario" refers to a practice or scene setting selected by the user in the dialogue simulator system.

[1429] A "terminal" is a hardware device (for example, a personal computer or smartphone) that allows a user to access and operate the dialogue simulator system.

[1430] The "server" is the central component of the dialogue simulator system, and is a computer that loads and customizes scenario data, generates AI characters, analyzes user speech data, and generates responses.

[1431] "Scenario data" refers to data such as templates, storylines, and example questions and answers related to each scenario in the dialogue simulator system.

[1432] The "database" is a data storage system for storing scenario data, the user's past practice history, dialogue history, feedback, and so on.

[1433] "Customization" is the process of adaptively modifying and adjusting a scenario, taking into account the user's past practice history and other relevant data.

[1434] An "AI character" is an artificial intelligence-based character (avatar) generated in a dialogue simulator system to engage in dialogue with a user.

[1435] "Character setting data" refers to data that includes information such as the AI ​​character's appearance, personality, intonation, and facial expression patterns.

[1436] "Utterance data" refers to recorded data of voice utterances made by a user during a conversation.

[1437] "Natural language processing" is a technology that allows an AI character to understand what the user is saying and generate an appropriate response based on that.

[1438] "Response data" is data that includes a response or reaction generated by the server in response to the content of the user's utterance.

[1439] "Feedback" is evaluation information that includes advice and suggestions for improving the dialogue, and is generated based on the user's speech content, facial expressions, intonation, etc.

[1440] "Dialogue history" refers to a record of the dialogues that a user has had with the dialogue simulator system.

[1441] The dialogue simulator system of the present invention begins when a user accesses the dialogue simulator system and selects a scenario. The user accesses the system from a browser on a PC or smartphone, logs in, and then selects a scenario that suits their purpose from a list of scenarios provided. For example, the user can select the scenario "Improve Presentation Skills."

[1442] Once a selection is made, the device sends this information to the server via an HTTP request (for example, the POST method). At this time, the data sent includes the user ID and the selected scenario ID. On the server side, the corresponding scenario data is read from a database (for example, MySQL or PostgreSQL) based on the received scenario ID. At the same time, the user's past practice history is obtained and the scenario is customized based on this. This customization reflects the themes the user has previously experienced and the points that should be emphasized.

[1443] Next, the server generates an AI character. This AI character is generated in real time using, for example, a custom Unity API. The generated character's appearance, personality, intonation, facial expression patterns, etc. are configured. This configuration data is sent to the device in JSON format, and the device uses WebGL or WebRTC technology to display it on the screen.

[1444] When a conversation begins, the device records the user's speech in real time and performs noise cancellation and echo removal to obtain high-quality audio data. This audio data is sent to a server, where it is converted into text using speech recognition technology such as the Google Cloud Speech-to-Text API. The server then analyzes the user's intent using natural language processing with a generative AI model (e.g., BERT or GPT-3). Based on the results of this analysis, the server generates an appropriate response and sends the response data back to the device.

[1445] Based on the received response data, the device uses voice synthesis technology such as the Web Speech API to have the AI ​​character respond, changing facial expressions and intonation along with the response to achieve a more natural dialogue.

[1446] Furthermore, the server analyzes the user's speech content, facial expressions, and speaking intonation in real time, and provides specific feedback based on the results. This feedback may include detailed advice such as, "The product explanation section was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more." This feedback is sent from the server to the device, which then provides it to the user. This process is carried out using emotion analysis tools such as Microsoft Azure's Emotion API.

[1447] Finally, the server stores all interaction history and feedback in a database, which can be used for future practice sessions, to analyze the user's progress, and to provide more customized scenarios.

[1448] Examples of specific examples and prompts

[1449] For example, if a user selects the "Improve Presentation Skills" scenario, the scenario will include the following prompt:

[1450] "Please tell us three features of your new product."

[1451] "Please suggest appropriate responses to the audience questions."

[1452] "Please tell me how to effectively summarize at the end of a presentation."

[1453] When a user says, "Today I'm going to give a presentation about our new product," the device's microphone records this speech and sends it to the server, which then responds with, "Please tell us more about your new product." After the presentation, the server analyzes the user's speech and provides feedback such as, "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more."

[1454] This system allows users to improve even the finer details of pronunciation, intonation, and facial expressions through dialogue.

[1455] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1456] Step 1:

[1457] This is the procedure in which a user accesses the dialogue simulator system and selects a scenario.

[1458] Input: User login information, system access, scenario selection request.

[1459] Data processing and calculation: User authentication, scenario list display, confirmation of selected scenario.

[1460] Output: Selected scenario ID, user ID.

[1461] Specific operation: A user logs in from a PC or smartphone browser, browses the list of scenarios, and then selects a scenario such as "Improve Presentation Skills."

[1462] Step 2:

[1463] This is a procedure in which the terminal transmits scenario selection information to the server.

[1464] Input: Selected scenario ID, user ID.

[1465] Data processing and calculation: Generating and sending HTTP requests (POST method, etc.).

[1466] Output: The scenario selection request sent to the server.

[1467] Specific operation: The device generates an HTTP request including the selected scenario ID and user ID and sends it to the server.

[1468] Step 3:

[1469] This is the procedure in which the server reads scenario data and the user's past practice history from the database and customizes the scenario.

[1470] Input: Scenario ID, User ID.

[1471] Data processing and calculation: Performing database queries, filtering and extracting past exercise history, and adaptive scenario customization.

[1472] Output: Customized scenario data.

[1473] Specific operation: The server reads the relevant scenario data and the user's past practice history from the MySQL database, and customizes the scenario based on the previous practice content.

[1474] Step 4:

[1475] This is the procedure in which the server generates an AI character and sends the character's setting data to the terminal.

[1476] Input: Customized scenario data.

[1477] Data processing and calculation: Calling the AI ​​character generation module and generating setting data.

[1478] Output: AI character configuration data (JSON format).

[1479] How it works: The server uses Unity's custom API to generate a character, create settings such as appearance, personality, intonation, and facial expression patterns, and send them to the device.

[1480] Step 5:

[1481] This is the procedure in which the device displays an AI character and begins a dialogue with the user.

[1482] Input: AI character configuration data.

[1483] Data processing and calculation: Reading setting data, using the screen drawing library.

[1484] Output: An AI character displayed on the screen.

[1485] Specific operation: The device displays an AI character on the screen based on the configuration data received using WebGL and WebRTC technology, and the conversation begins when the user presses the "Start" button.

[1486] Step 6:

[1487] This is a procedure in which the terminal records the user's speech data and transmits it to the server.

[1488] Input: User utterance.

[1489] Data processing and calculation: voice recording, noise cancellation, echo removal.

[1490] Output: Preprocessed audio data.

[1491] Specific operation: When a user speaks to an AI character, the device's microphone records the voice, performs noise cancellation and echo removal in real time, and then sends the voice data to the server.

[1492] Step 7:

[1493] This is the procedure where the server analyzes the voice data and generates an appropriate response.

[1494] Input: Preprocessed audio data.

[1495] Data processing and calculation: Speech-to-Text, natural language processing (BERT and GPT-3).

[1496] Output: Response data (text format).

[1497] How it works: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text, then uses a natural language processing model to analyze the user's intent and generate an appropriate response.

[1498] Step 8:

[1499] This is the procedure by which the device displays a response to the user through an AI character.

[1500] Input: Response data from the server (text format).

[1501] Data processing and calculation: voice synthesis, facial animation generation.

[1502] Output: Voice responses and facial expressions by the AI ​​character.

[1503] How it works: The device uses the Web Speech API to convert the response into voice and responds to the user along with the AI ​​character's facial expressions, resulting in a more natural dialogue.

[1504] Step 9:

[1505] This is a procedure in which the server analyzes the user's speech and generates real-time feedback.

[1506] Input: User utterance and parsed results.

[1507] Data processing and calculation: Sentiment analysis, speech evaluation, and extraction of areas for improvement.

[1508] Output: Feedback data.

[1509] Specific operation: The server uses Microsoft Azure's Emotion API and emotion analysis tools to evaluate the user's speech and generate feedback including specific improvements and advice.

[1510] Step 10:

[1511] This is a procedure in which the terminal provides feedback to the user.

[1512] Input: Feedback data.

[1513] Data processing and calculation: Display of feedback contents.

[1514] Output: The displayed feedback.

[1515] Specific operation: The device provides the received feedback data to the user by voice or text, using a text box displayed on the screen or a speaker for voice feedback.

[1516] Step 11:

[1517] This is the procedure by which the server stores the dialogue history and feedback in a database.

[1518] Input: Dialogue history, feedback data.

[1519] Data processing and calculation: Writing to database, saving data.

[1520] Output: Stored interaction history and feedback.

[1521] What it does: The server stores all interaction history and feedback in a MySQL database, which is used to customize the scenario for the next session.

[1522] (Application example 1)

[1523] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1524] Previously, training to improve customer service skills in brick-and-mortar stores required effort and cost to deal with real customers, and had the problem of not being able to receive immediate feedback. Furthermore, there was a lack of opportunities for new staff to practice interacting with real customers, making it difficult for them to improve their skills. For this reason, there is a need for a system that can improve customer service skills while receiving effective, real-time feedback.

[1525] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1526] In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and send character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and send it to the server; means for the server to analyze the user's utterance data to generate an appropriate response and send it to the terminal; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation and generate feedback; means for the server to send the generated feedback to the terminal and provide it to the user; means for the server to save the user's dialogue history and feedback and use it for future learning; means for the terminal to function as a customer service simulator and enable brick-and-mortar store staff to improve their customer service skills; and means for providing interactive feedback to the user. This enables brick-and-mortar store staff to improve their customer service skills by receiving feedback effectively and in real time.

[1527] A "dialogue simulator system" is a system that allows users to improve specific skills or knowledge by engaging in dialogue in a simulated format.

[1528] A "scenario" is a set of stories or conditions designed to simulate a particular situation or dialogue.

[1529] A "server" is a computer system that serves as the core of the dialogue simulator system accessed by users, and is a device that processes data and generates AI characters.

[1530] "Means for reading data and customizing a scenario" refers to a mechanism in which the server reads related information from a database for a scenario selected by a user and modifies it to suit the user's needs.

[1531] An "AI character" is a virtual character with artificial intelligence that is generated in a dialogue simulator to have an interactive conversation with a user.

[1532] "Configuration data" is data that defines the specific attributes and behavior of an AI character, such as their appearance, personality, intonation, and facial expression patterns.

[1533] "Utterance data" is data of the voice uttered by the user during a conversation.

[1534] A "reply" is a message that the server generates in response to an analysis of the user's speech data.

[1535] "Expressions" refer to the facial movements and emotional expressions shown by AI characters in the dialogue simulator.

[1536] "Intonation" refers to the rise and fall of the voice when speaking.

[1537] "Feedback" refers to information such as evaluations and areas for improvement provided based on the user's interactions and actions.

[1538] "Dialogue history" is a record of past dialogues between the user and the AI ​​character.

[1539] The "customer service simulator" is a system that allows store staff to simulate interactions with customers and improve their response skills.

[1540] "Interactive feedback" refers to dynamic evaluations and advice provided in real time in response to a user's actions.

[1541] This invention relates to a customer interaction simulator system that allows store staff to improve their customer interaction skills. The system starts when a user accesses the interaction simulator system and selects a scenario. Based on the selected scenario, the server loads data and customizes the scenario. The server then generates an AI character and sends its setting data to the terminal.

[1542] After the user initiates a dialogue, the device receives the user's speech data and sends it to the server. The server analyzes the user's speech data, generates an appropriate response using a generative AI model, and sends this response data to the device. The device then provides a response to the user through an AI character. At this time, the AI ​​character's facial expression and intonation also change according to the scenario. The server then analyzes the content of the user's speech, facial expression, and speaking intonation to generate feedback. The generated feedback is sent to the device and provided to the user.

[1543] Hardware and software used

[1544] Hardware:

[1545] Smart glasses (e.g., smart devices)

[1546] Head-mounted displays (e.g., mixed reality devices)

[1547] software:

[1548] Python (programming language)

[1549] speech_recognition (speech recognition library)

[1550] transformers (natural language processing library)

[1551] The server captures the user's voice input data and converts it into text using a speech recognition library. The converted data is then used by an AI character to generate an appropriate response using a natural language processing library. This response is then provided to the user via the device. The user's conversation is analyzed in real time and feedback is generated. This feedback includes specific advice and suggestions for improvement and is provided to the user interactively via the device.

[1552] Specific examples

[1553] For example, suppose a staff member at a brick-and-mortar store selects the "Basic Customer Response Scenario" and starts training. The staff member says, "Hello, is there anything you'd like to buy in our store?" The device records this speech, and the server analyzes the audio data. Based on the analysis results, the server generates a response such as "Hello, is there anything in particular you'd like to buy today?" and displays the response through the device.

[1554] The system then analyzes the speech and intonation of the staff member and provides feedback such as, "If you ask your question with a more emphasized intonation, it will be easier for the customer to understand." This feedback is provided interactively in real time, helping the staff member improve their skills.

[1555] In this way, store staff can effectively improve their customer service skills.

[1556] Example prompt sentence:

[1557] "Hello, are you looking for something in the store?"

[1558] "Is there anything in particular you're looking for today?"

[1559] "If you ask questions with a bit more emphasis on intonation, you'll be able to communicate more clearly to the customer."

[1560] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1561] Step 1:

[1562] A user accesses the dialogue simulator system and selects a scenario. As input, the user accesses the scenario selection interface from the device they use (smart glasses or a head-mounted display) and selects a specific scenario (e.g., "Basic Customer Interaction Scenario"). As output, the selected scenario information is sent to the server.

[1563] Step 2:

[1564] The server reads the data corresponding to the selected scenario and customizes the scenario. The input is the scenario selection information sent by the user, and based on this the server retrieves related information from the database. The data is processed by customizing the scenario based on the user's past practice history, and customized scenario data is generated as output.

[1565] Step 3:

[1566] The server generates an AI character and sends the character's configuration data to the device. The input is customized scenario data, and the AI ​​character's configuration (appearance, personality, intonation, facial expression patterns, etc.) is generated. The data is processed using a generative AI model to generate the character, and the configuration data is generated as output and sent to the device.

[1567] Step 4:

[1568] The user starts a conversation. The device receives the user's speech data and sends it to the server. The input is the user's voice data at the start of the conversation, which the device captures in real time. The data is then processed using a speech recognition library to convert the voice data into text data, and the converted text data is generated as output and sent to the server.

[1569] Step 5:

[1570] The server analyzes the user's speech data, generates an appropriate response, and sends it to the device. The converted text data is used as input, and a response is generated using a generative AI model based on this. Natural language processing is used to process the data, analyzing the user's intent, and an appropriate response data is generated as output and sent to the device.

[1571] Step 6:

[1572] The device provides the user with a response from an AI character. The response data sent from the server is input, and the device receives this. The data is processed by changing the AI ​​character's facial expression and intonation based on the settings, and the response is provided to the user visually and audibly as output.

[1573] Step 7:

[1574] The server analyzes the user's speech content, facial expressions, and speaking intonation to generate feedback. The input is the user's latest speech data and dialogue history, and data calculations are performed using an AI model based on this data. The output is feedback that includes specific advice and suggestions for improvement.

[1575] Step 8:

[1576] The server sends the generated feedback to the terminal and provides it to the user. The generated feedback data is input and sent to the terminal. The data is processed by adjusting the display format of the feedback, and the feedback is provided to the user in an interactive form as output.

[1577] Step 9:

[1578] The server stores the user's interaction history and feedback and uses it for future learning. The input is the latest interaction history and feedback data, and the data is processed and stored appropriately in a database. The output is used to customize future scenarios and improve the learning algorithm.

[1579] Example prompt sentence:

[1580] "Hello, are you looking for something in the store?"

[1581] "Is there anything in particular you're looking for today?"

[1582] "If you ask questions with a bit more emphasis on intonation, you'll be able to communicate more clearly to the customer."

[1583] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1584] The dialogue simulator system of the present invention is structured so that a user accesses the system and selects a scenario, and the entire system is processed mainly by the server. The specific flow of processing will be explained below.

[1585] When starting to use the system, the user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface. The user selects the desired scenario (e.g., sales role-play, meeting practice, presentation skill improvement, etc.), and the selection information is sent from the terminal to the server.

[1586] The server reads data corresponding to the selected scenario from the database and customizes the scenario based on the user's past practice history. Based on this customized scenario, the server generates configuration data for the AI ​​character and sends the configuration data to the device. The device displays the generated AI character to the user, ready for the user to begin interacting with it.

[1587] When a user initiates a dialogue and speaks, the device records the user's voice input and sends it to the server in real time. The server analyzes the voice data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends the response data to the device. The device responds to the user by changing the AI ​​character's facial expression and intonation.

[1588] Furthermore, the system uses an emotion engine to analyze the user's speech and facial expression data. The server uses the emotion engine to recognize the user's emotions and adjusts the response and feedback accordingly. For example, if the user is nervous, the AI ​​character will respond in an encouraging manner to calm the user.

[1589] The server also uses the emotion engine to generate feedback. It analyzes the user's speech content, facial expressions, and speaking intonation in real time, and generates specific feedback based on the analysis results. This feedback is sent to the device and provided to the user in the form of voice or text. The feedback includes specific advice and areas for improvement.

[1590] For example, suppose a user selects a scenario for improving presentation skills and begins a presentation. When the user says, "Today I'm going to give a presentation about our new product," the device records this speech and sends it to the server. The server analyzes this speech data and generates a response such as, "Tell me more about your new product." It then analyzes the user's emotions using an emotion engine and adjusts the tone and content of the response as needed.

[1591] After the presentation, the server analyzes the user's speech and intonation, and provides specific feedback such as, "The product explanation was particularly easy to understand in the presentation. However, it would be even better if you emphasized the intonation a little more." The feedback is also adjusted appropriately, taking into account the user's emotions.

[1592] Finally, the server stores the user's dialogue history and feedback in a database and uses it as data to personalize the next scenario. In this way, the dialogue simulator system of the present invention helps users improve their skills and provides a more effective training environment.

[1593] To implement the invention, a program is required to smoothly carry out a series of operations, including user selection, data analysis, emotion recognition, and feedback generation, allowing users to effectively improve their practical skills through realistic interactions.

[1594] The processing flow will be explained below.

[1595] Step 1:

[1596] The user accesses the dialogue simulator system and checks the scenario selection menu on the terminal interface.

[1597] Step 2:

[1598] The user selects the scenario of their choice (e.g., sales role-play, meeting practice, presentation skill improvement, etc.).

[1599] Step 3:

[1600] The terminal transmits the user's scenario selection information to the server.

[1601] Step 4:

[1602] The server reads data corresponding to the selected scenario from the database.

[1603] Step 5:

[1604] The server reviews the user's past practice history and customizes the scenario, which can include adjusting the practice level or focusing on specific tasks.

[1605] Step 6:

[1606] The server generates the AI ​​character's configuration data, which includes the character's appearance, personality, intonation, and facial expression patterns generated by the emotion engine.

[1607] Step 7:

[1608] The server sends the AI ​​character configuration data generated by the server to the device.

[1609] Step 8:

[1610] The device displays the AI ​​character to the user and prepares to begin a conversation.

[1611] Step 9:

[1612] The user initiates the interaction and speaks the first word (e.g., beginning a presentation).

[1613] Step 10:

[1614] The device records the user's voice input in real time and sends the data to the server.

[1615] Step 11:

[1616] The server receives the recorded data, performs voice analysis, and analyzes the user's linguistic intent.

[1617] Step 12:

[1618] The server uses an emotion engine to analyze the facial expression data collected at the same time as the user's speech data, and recognizes the user's emotions.

[1619] Step 13:

[1620] The server generates an appropriate response based on the user's intentions and emotions, and adjusts the character's facial expression and intonation.

[1621] Step 14:

[1622] The server sends the generated response data to the terminal.

[1623] Step 15:

[1624] The device changes the AI ​​character's facial expression and intonation to provide responses to the user in voice or text.

[1625] Step 16:

[1626] The user makes the next utterance and the dialogue proceeds. The device again records the user's voice input and sends it to the server.

[1627] Step 17:

[1628] The server analyzes the user's continuous speech data and facial expression data, and generates feedback in real time based on the user's emotions and speech content.

[1629] Step 18:

[1630] The server transmits the generated feedback data to the terminal.

[1631] Step 19:

[1632] The device provides real-time feedback to the user via voice or text, providing suggestions for improvement and advice.

[1633] Step 20:

[1634] The server stores the user's interaction history and feedback in a database.

[1635] Step 21:

[1636] The server analyzes the user's learning status based on the stored data and generates data to customize the content of the next practice session.

[1637] Step 22:

[1638] The device will then display a report of the results to the user, including a score for the exercise, specific areas for improvement, and advice for the next exercise.

[1639] Through these detailed steps, users can effectively improve their skills while experiencing realistic interactions. By utilizing the emotion engine, appropriate responses and feedback are provided that are tailored to the user's emotions, creating a more natural and effective learning environment.

[1640] Example 2

[1641] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1642] Conventional dialogue simulator systems lack real-time feedback on user voice input and dialogue content, making it difficult to customize specific scenarios that effectively reflect the user's emotions and practice history.Furthermore, it is difficult to provide a realistic dialogue environment by changing facial expressions and intonation during dialogue.

[1643] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and transmit character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and transmit it to the server; means for the server to analyze the user's utterance data, generate an appropriate response, and transmit it to the terminal; means for the terminal to respond while changing the AI ​​character's facial expression and intonation; means for the server to recognize the user's emotions using an emotion engine and adjust the response content and feedback based on the recognition; means for transmitting the feedback generated by the server to the terminal and providing it to the user; and means for the server to save the user's dialogue history and feedback and use it for future learning. This enables advanced scenario customization that reflects the user's practice history and emotions and feedback with real-time emotion recognition.

[1644] A "simulator system" is a computer-based system for conducting interactive training for a user.

[1645] A "user" is an individual who uses this system to select a scenario and conduct dialogue training.

[1646] A "server" is a computer that centrally processes the entire system, analyzes input data from users, and generates appropriate scenarios and responses.

[1647] A "terminal" is a device through which a user accesses the system and uses the functions of the dialogue simulator system.

[1648] A "scenario" refers to a specific situation or setting prepared for dialogue training.

[1649] A "database" is a system for managing and storing data used by the dialogue simulator system.

[1650] "Customization" refers to adjusting and changing scenarios and settings based on a user's past history and specific needs.

[1651] An "AI character" is a character with artificial intelligence that converses with the user within the dialogue simulator system.

[1652] "Configuration data" is information used to determine the appearance, tone of voice, and behavior of an AI character.

[1653] "Utterance data" is data of the voice uttered by the user during a conversation.

[1654] "Natural language processing" is a technology that enables computers to understand and generate human language.

[1655] An "emotion engine" is software that analyzes a user's voice and facial expressions to determine their emotional state.

[1656] "Feedback" refers to evaluations and advice given to a user's interactions and actions.

[1657] The "dialogue history" is a record of the content of dialogues and scenarios that the user has had in the past.

[1658] "Intonation" refers to the intonation and pitch of the voice when speaking.

[1659] "Facial expressions" are expressions on the user's face that indicate emotions and reactions.

[1660] The dialogue simulator system of the present invention is structured so that the user accesses the system and selects a scenario, and the entire system is processed by a server. The system analyzes the user's speech data, facial expression data, and dialogue history to provide an environment for advanced dialogue with the user.

[1661] The specific hardware required to implement this system includes the devices used by users (e.g., PCs, smartphones, tablets) and the server that controls the entire system. The main software used includes a speech recognition API (e.g., Google Cloud Speech-to-Text API), a natural language processing module, an emotion recognition API (e.g., Microsoft Azure Cognitive Services Emotion API), and a module for AI character generation.

[1662] When starting to use the system, the user accesses the dialogue simulator system through their own device and checks the scenario selection menu on the interface. When the user selects the desired scenario, the selection information is sent from the device to the server. The server reads data based on the selected scenario from the database and customizes the scenario by referring to the user's past practice history. Based on the customized scenario, the server generates configuration data for the AI ​​character and sends it to the device.

[1663] When a user initiates a dialogue, the device records the user's voice input in real time and sends it to the server. The server analyzes the voice data and uses natural language processing to understand the user's intent. The server then generates an appropriate response and sends the response data to the device. The device responds to the user by changing the AI ​​character's facial expression and intonation.

[1664] In addition, the server uses an emotion engine to recognize the user's emotions and adjusts responses and feedback accordingly. For example, if the user is nervous, the AI ​​character will respond in an encouraging manner to ease the user's tension. The server also generates specific feedback based on the results of analyzing the user's speech and facial expression data and sends it to the device. The generated feedback is provided to the user in the form of voice or text, and includes specific advice and areas for improvement.

[1665] Furthermore, the server stores the user's dialogue history and feedback in a database. This stored data is used as personalized data for the next scenario customization, allowing the user to continuously receive dialogue training that is individually adapted to their needs.

[1666] As a concrete example, let's say a user selects the "Improve Presentation Skills" scenario and begins a presentation. When the user says, "Today I'm going to give a presentation about our new product," the device records this speech and sends it to the server. The server analyzes the audio data and generates a response such as, "Tell me more about your new product." At this time, it analyzes the user's emotions using an emotion engine and adjusts the tone and content of the response as necessary. After the presentation ends, the server analyzes the content and intonation of the user's speech and provides specific feedback such as, "The part of your presentation that was particularly easy to understand was the product explanation. However, it would be even better if you emphasized your intonation a little more."

[1667] An example of a prompt in this system is, "You have selected a scenario for improving your presentation skills and have begun your presentation. Please say, 'Today I will be presenting about our new product.'"

[1668] This allows the dialogue simulator system to help users improve their skills and provide a realistic practice environment.

[1669] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1670] Step 1:

[1671] The user accesses the dialogue simulator system and selects a scenario.

[1672] Specific operation: The user opens the browser on the device, accesses the system URL, and selects "Improve Presentation Skills" from the scenario menu.

[1673] Input: Scenario information selected by the user when they access the system.

[1674] Output: The selected scenario information is sent from the terminal to the server.

[1675] Step 2:

[1676] The server loads data corresponding to the selected scenario and customizes the scenario.

[1677] Specific operation: The server loads the "Improve Presentation Skills" scenario template from the database. It then refers to the user's past history and adjusts and modifies the scenario.

[1678] Input: Selected scenario information and user practice history data.

[1679] Output: Customized scenario data.

[1680] Step 3:

[1681] The server generates configuration data for the AI ​​character and sends it to the device.

[1682] Specific operation: Set the AI ​​character's name, appearance, tone of voice, etc., and send the setting data to the device in JSON format.

[1683] Input: Customized scenario data.

[1684] Output: AI character configuration data is sent to the device.

[1685] Step 4:

[1686] The user starts a dialogue, and the terminal receives the utterance data and transmits it to the server.

[1687] Specific operation: The user says, "Today I will be giving a presentation about our new product." The device records the voice and sends it to the server.

[1688] Input: User's voice input.

[1689] Output: The recorded audio data is sent to the server.

[1690] Step 5:

[1691] The server analyzes the user's voice data, generates an appropriate response, and sends it to the terminal.

[1692] Specific operation: The voice data is converted to text using the Google Cloud Speech-to-Text API, and the text data is analyzed using a natural language processing module to generate a response.

[1693] Input: Recorded audio data.

[1694] Output: Response data such as "Please tell me more about your new product" is generated and sent to the terminal.

[1695] Step 6:

[1696] The device responds by changing the AI ​​character's facial expression and intonation.

[1697] Specific operation: The text response is converted into speech using a speech synthesis API, and the response is played back as speech while the AI ​​character's facial expression changes.

[1698] Input: Response data.

[1699] Output: A voice-synthesized response and facial expression changes are provided to the user.

[1700] Step 7:

[1701] The server uses an emotion engine to recognize the user's emotions and adjusts responses and feedback accordingly.

[1702] Specific behavior: Analyzes emotions using the emotion engine (Microsoft Azure Cognitive Services Emotion API) and adjusts the tone and content of responses.

[1703] Input: User's voice and facial expression data.

[1704] Output: The adjusted response data.

[1705] Step 8:

[1706] The server generates feedback and sends it to the terminal for presentation to the user.

[1707] Specific actions: Analyze the user's speech content and intonation and generate specific feedback, such as, "The product explanation was particularly easy to understand in your presentation. However, it would be even better if you emphasized your intonation a little more."

[1708] Input: Speech content, intonation, and facial expression data.

[1709] Output: The generated feedback data is sent to the terminal and provided to the user.

[1710] Step 9:

[1711] The server stores the user's interaction history and feedback and uses it for future learning.

[1712] Specific operation: After the dialogue session ends, the spoken content and emotional data are saved in a database. The next time a scenario is customized, an individually adapted scenario is generated based on the saved data.

[1713] Input: User interaction history and feedback data.

[1714] Output: Stored interaction history and feedback data.

[1715] (Application example 2)

[1716] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1717] In today's brick-and-mortar stores, improving the customer service skills of store associates is important, but there is a lack of systems for effectively training them. Furthermore, traditional training methods lack consistency and objectivity, making it difficult to provide individualized instruction tailored to each employee's strengths and weaknesses.

[1718] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to access the dialogue simulator system and select a scenario; means for the server to read data corresponding to the selected scenario and customize the scenario; means for the server to generate an AI character and send character setting data to the terminal; means for the terminal to receive utterance data for the user to start a dialogue and send it to the server; means for the server to analyze the user's utterance data, generate an appropriate response, and send it to the terminal; means for the server to analyze the user's utterance content, facial expressions, and speaking intonation and generate feedback; means for the server to send the generated feedback to the terminal and provide it to the user; means for the server to save the user's dialogue history and feedback and use it for future learning; means for providing customer service training scenarios to the terminal, including a function to perform emotion analysis in real time and adjust responses; and means for the terminal to present detailed feedback and areas for improvement after the training is completed. This enables consistent, high-quality customer service training and effectively supports employee skill improvement.

[1719] A "dialogue simulator system" is a system in which the user selects a scenario and engages in dialogue with an AI character based on that scenario.

[1720] A "scenario" is a set situation or theme for a user to practice with the dialogue simulator.

[1721] The "server" is a central processing unit that processes the scenarios and speech data selected by the user, generates AI characters, and provides feedback.

[1722] An "AI character" is a virtual character equipped with artificial intelligence that is generated in a dialogue simulator system to engage in dialogue with a user.

[1723] A "terminal" is a computer device that allows a user to access the dialogue simulator system and select a scenario or engage in dialogue.

[1724] "Utterance data" is voice data of what the user says in the dialogue simulator system.

[1725] "Feedback" is information indicating advice and areas for improvement provided based on the analysis of the user's dialogue content, facial expressions, and speaking style.

[1726] "Emotion analysis" is a technology that analyzes a user's emotional state from their speech and facial expressions.

[1727] A "customer service training scenario" is a specific interaction situation designed to help store staff improve their customer service skills.

[1728] "History" is a record of the interactions a user has had and feedback they have received in the past.

[1729] "Consistency" means that customer service training using the dialogue simulator system is always conducted with the same quality.

[1730] "High-quality customer service training" refers to providing effective training to improve customer service skills that is tailored to the strengths and weaknesses of each individual store employee.

[1731] The embodiment of the present invention is aimed at "customer service training" and specifically includes a step in which a user utilizes a dialogue simulator system to improve customer service skills.

[1732] First, a user accesses the dialogue simulator system using a device (e.g., a smartphone) and selects a specific scenario for training (e.g., introducing a new product or handling a complaint). The device then sends the selection information to the server, which then reads the scenario data from a database and customizes the scenario based on the user's past practice history.

[1733] Next, the server generates an AI character based on the customized scenario and sends the character's configuration data to the device. The device then prepares for the user to begin a dialogue, and the user speaks to simulate a real dialogue. For example, a user who selected a new product introduction scenario might say, "Tell me about the new products that have just arrived."

[1734] As the conversation progresses, the device sends the user's speech data in real time to the server, which then analyzes it using a natural language processing (NLP) engine. Specifically, it uses the Google Cloud Natural Language API to understand what the user is saying and generate appropriate responses based on that. This process also utilizes generative AI models.

[1735] In addition to generating responses, the server uses Affectiva's emotion recognition engine to analyze the user's emotional state and adjust the AI ​​character's facial expressions and intonation accordingly. The server also generates detailed feedback based on the analysis of the speech and facial expression data, suggesting areas for improvement to the user. This feedback includes specific advice and areas for improving effective customer service skills. After training is complete, the feedback is returned to the terminal and presented to the user.

[1736] The server stores this dialogue history and feedback in a database and uses it as data to make the next training session more effective.The next time a scenario is selected, it is automatically customized based on past history, providing the user with the optimal training environment.

[1737] For example, if a user says, "Today I'm going to give a presentation about our new product," the server will respond with, "Tell me more about your new product." If the user is nervous, the AI ​​character will offer words of encouragement, such as, "Relax and let's talk. Let's do our best together."

[1738] Examples of prompts:

[1739] "New product introduction"

[1740] User: "Tell me about the new items you've got in stock."

[1741] AI Character: "This new product uses the latest technology and offers 50% longer battery life than the previous model. Plus, it has a new, stylish design."

[1742] In this way, the dialogue simulator system effectively supports the improvement of customer service skills in brick-and-mortar stores, enabling store clerks to provide high-quality service in actual customer service situations.

[1743] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1744] Step 1:

[1745] A user accesses a terminal and starts the dialogue simulator system. A scenario selection menu is displayed so that the user can select a specific customer service training scenario.

[1746] Input: User's scenario selection

[1747] Output: Selected scenario information

[1748] Specific operation: The terminal acquires the scenario information selected by the user from the menu, formats the selected information, and sends it to the server.

[1749] Step 2:

[1750] Based on the scenario selection information received by the server, the server reads the corresponding scenario data from the database and customizes the scenario based on the user's past practice history.

[1751] Input: Selected scenario information, user's past practice history

[1752] Output: Customized scenario data

[1753] Specific operation: The server queries the database to obtain the user's practice history and selected scenario data. Based on this data, it customizes the scenario using scripts and a rules engine.

[1754] Step 3:

[1755] The server generates an AI character based on the customized scenario data and sends the character's setting data to the device.

[1756] Input: Customized scenario data

[1757] Output: AI character configuration data

[1758] Specific operation: The generative AI model is used to generate dialogue and facial expression data for the AI ​​character, and the setting data is sent to the device.

[1759] Step 4:

[1760] The user starts a conversation with the AI ​​character on their device. The user's speech is recorded by the device's microphone and sent to the server in real time.

[1761] Input: User utterance

[1762] Output: Sending speech data to the server

[1763] Specific operation: The device records the user's speech as audio data, converts it into an appropriate format, and sends it to the server.

[1764] Step 5:

[1765] The server analyzes the user's speech data using a natural language processing engine (Google Cloud Natural Language API) and generates an appropriate response.

[1766] Input: User utterance data

[1767] Output: The generated response

[1768] How it works: The server inputs the speech data into a natural language processing engine, analyzes the user's intent, and uses a generative AI model to generate an appropriate response.

[1769] Step 6:

[1770] The server analyzes the user's speech and facial expression data using an emotion analysis engine (Affectiva) and adjusts the AI ​​character's facial expressions and intonation.

[1771] Input: Speech data, facial expression data

[1772] Output: Adjusted response data

[1773] How it works: The server inputs speech data and facial expression data into an emotion analysis engine to identify the user's emotional state, and then adjusts the AI ​​character's facial expression and intonation accordingly.

[1774] Step 7:

[1775] The server sends the response and analysis results to the terminal, which displays them to the user.

[1776] Input: Adjusted response data

[1777] Output: The response that is displayed to the user

[1778] Specific operation: Based on the response data received from the server, the device updates the display of the AI ​​character and provides a response to the user along with audio.

[1779] Step 8:

[1780] After the training is completed, the server analyzes the user's interaction history and generates detailed feedback and suggestions for improvement.

[1781] Input: Dialogue history

[1782] Output: Detailed feedback

[1783] How it works: The server analyzes data from completed training sessions and uses a generative AI model to generate feedback, including analysis of the user's speech and facial expressions.

[1784] Step 9:

[1785] The server generates feedback and sends it to the terminal for presentation to the user.

[1786] Input: Detailed feedback

[1787] Output: Feedback presented to the user

[1788] Specific operation: The device acquires feedback data and provides it to the user in text or voice.

[1789] Step 10:

[1790] The server stores the interaction history and feedback in a database for use in future training sessions.

[1791] Input: Dialogue history, feedback

[1792] Output: Saved data

[1793] Specific operation: The server stores the dialogue history and feedback in a database, and refers to this the next time a scenario is selected to customize it.

[1794] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1795] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1796] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1797] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1798] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1799] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1800] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1801] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1802] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1803] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1804] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1805] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1806] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1807] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1808] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1809] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1810] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1811] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1812] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1813] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology o...

Claims

1. a means for a user to access the dialogue simulator system and select a scenario; a means for the server to read data corresponding to the selected scenario and customize the scenario; A means for the server to generate an AI character and send the character's setting data to the terminal; a means for receiving speech data for a user to start a dialogue and transmitting the data to a server; A means for the server to analyze the user's speech data, generate an appropriate response, and transmit the response to the terminal; A means for the server to analyze the content of the user's speech, facial expressions, and speaking intonation and generate feedback; means for transmitting server-generated feedback to the terminal for providing to the user; A means for the server to store the user's interaction history and feedback and use it for future learning; This system provides a dialogue simulator that includes facial expressions and intonation.

2. 2. The system according to claim 1, wherein the terminal includes means for changing the facial expression and intonation of the AI ​​character while replying.

3. The system according to claim 1, wherein the server includes means for customizing a scenario based on the user's past practice history.

Images